Predictive Maintenance Using Deep Learning and Reliability Engineering with Shayan Mortazavi - #540

Key Takeaways

[Music] all right everyone i am here with cheyenne mortazavi cheyenne is a data science manager at accenture and cheyenne i'm excited to welcome you on to the podcast oh wow sam it's pleasure to be here i'm really thrilled to to talk to you and yeah looking forward to it let's do it so you recently presented at sig ops ai and hpc summit and we're going to dig into your presentation and some of the work that you're doing there at accenture but to get us started why don't you share a little bit about your background and how you came to work in data science sure well um i'm an engineer by background and i started as a mechanical engineer in energy domain and building structures and pressurized systems in energy sector oil and gas industry and then i moved slightly into simulation of the structures and the components that are used mostly in the heavy industry domain and then i got interested into material science and worked a little bit conjunction between the simulation software's built for analyzing basically structures in that domain and material in terms of their characteristic the properties and then through that i had to work a lot with the with data and doing a lot of tests and these structures and uh with their properties and making sure that they are fit for their service and uh good integrity and so dealing with a lot of data kind of got me to think about are there other applications uh for this data that we receive which in the engineering domain it's usually people working with these formulas and this scary calculation chits and softwares and simulations so yeah and then i got excited about using data to solve some of the difficult problems that exist in engineering domain slowly slowly moved into machine learning and statistical learning and then yeah nice nice well tell us a little bit about your group at accenture and your role there when i joined accenture i joined a group called industrial analytics group and it is a group which is primarily focusing on application machine learning and statistical influence and heavy industry and resources what we call kind of covering mining sector and oil and gas renewables every every sort of manufacturing base that have some asset heavy heavy assets and then yeah in there we kind of look into multiple different type of problems from production efficiency supply chain predicting maintenance and optimization of the the production plants and yeah these these sort of things got it and predictive maintenance which you mentioned was the topic that you focused on uh for your talk at the sigops summit can you give us a little background on [Music] predictive maintenance and in particular some of the the use cases that you've worked on well that's an interesting topic because especially in oil and gas and energy sector because they are heavily regulated and there are you know decades of applying best practices into their operations and their maintenance systems it is kind of uh there is a inertia and [Music] changing or shifting the to towards more advanced sort of solutions um [Music] so maintenance itself uh as a problem is trying to maximize or increase the reliability of assets and this can be cars it could be trucks could be stationary static sort of equipment tanks and towers and drums could be some rotating machineries so if the objective is to maximize the reliability increase their availability and efficiency um there's been traditionally kind of a myriad of strategies that asset operators depends on complexity of assets and they have applied um traditional practices like fixed based maintenance where you let the machine run to failure and then you fix when it is broken so it's very costly and you don't have any view what is the state of the machine and then um there are some kind of more clever um more um systematic sort of approaches in the maintenance though called time-based maintenance or scheduled maintenance where you define a sort of a strategy of how to um regularly kind of getting a sense for the meantime the failure of different components and using that to build up to exactly when the thing might break and absolutely yeah these sort of things um so in the past let's say three four decades uh when iot and scott assistant came into heavy industry domain then a branch of let's say preventative maintenance emerge called condition-based monitoring where you the objective is to monitor the health of the equipment at any level uh using the sensor sensory data receive and then um use some early warning sort of functionality on top of that to detect any problems at system level or different hierarchy and then act on top of that so preventative maintenance under the condition based monitoring these cbms are kind of resolved a lot of the traditional issues in the in this domain and predictive maintenance is a child of the cbm systems like the condition-based monitorings which simply the objective is can we use a fully data-driven system to predict when assets are going to fail in future and based on that optimize their maintenance actions and optimize over the resources and costs so yeah and so your presentation uh long title a novel framework for predictive maintenance using deep learning and reliability engineering uh the your the so the kind of the broad context or problem that you're working on is predictive maintenance um what was the specific use case that gave rise to uh some of the work that you talked about beautiful yeah so in this work we tried to not just fully rely on a data driven machine learning sort of approach but see how we can combine and marry the the domain knowledge uh exists for decades of being built up uh with the sort of capabilities we get from machine learning uh to address this very specific problem of can we detect failures at different levels of these machineries in advance and provide some insights that can be meaningfully taken as maintenance actions in order to move away from like reactive sort of maintenance or and also to let them to judge whether they can delay the maintenance actions or interventions or shutdown of their plans so this framework was specifically trying to break down the systems these complex systems uh into multiple different levels in terms of the complexity and the functionality in the system uh in the entire system from reliability point of view and then use machine learning to kind of monitor the performance of these little building blocks and then uh by by the means of aggregation through the science of reliability engineering then make sense of what is the global picture so this framework specifically yes was applied into rotating machineries which we use everywhere in the petrochemical plants and power plants and refineries pharmaceutical industry everywhere and likes generators turbo turbo generators pumps compressors and turbines these type of things exactly so these are complex these machineries are um you can look at them like the turbo generators as jet engines they they have a very high rpm uh rotating parts uh like 20 000 to 30 000 sometimes rpm like exactly like a jet engine lots of moving parts high vibration under significant loads temperature pressure and yeah heavily sensor yeah i don't remember the specific stat but you know a few years ago when in the early days of iot like there was this boeing jet engine or ge aircraft engine example that we would throw around all the time and just how many sensors were in this engine and it was an astounding number um and so this is all um time series data that you're collecting so this is an interesting point because these these machines are let's say in the energy sector they are heavily censored up to 2000 3000 sometimes to a number of sensors monitoring um like pressure temperature vibration flow rate some process parameters but the purpose of these sensors are not necessarily to provide maintenance inputs into maintenance engineers that are there to for safety purposes um or to regulate the process um so the and in order to address a problem of maintenance we don't necessarily have the adequate type location of the sensors in terms of um the type of time series that we want or the parameters we want to monitor uh and also the redundancy in the type of sensors that we want to have to monitor specific modes of failures for example and also there is no scope for retrofitting type of sensors onto these complex machines um so yeah we receive a lot of time series data and we have to apply a predictive solution which is a kind of new generation approach on top of old-fashioned censoring sort of practices and that makes it a little bit interesting that it's really interesting that you put it like that the idea that the sensors that are in these machines aren't necessarily there for you to make predictions from that that it strikes me that that was always left out of this narrative of like iot and predictive maintenance and digital twins that i've heard over and over and over again but uh what you're describing is like you've got all these data points that are being spun out of these machines you've got a bunch of you know many many years of equations to predict the reliability of the machines and one you might need to repair them the whole field of reliability engineering that's evolved but there's a gap it's like the bridge was built from two shores and doesn't meet in the middle and it sounds like that's what the core of your presentation was you're using predictive models to map from the data you're collecting off of these sensors to the equations that you would traditionally use precisely that's that's a very good uh summary of yeah and of the problem itself yeah and maybe going forward into future where operators and the manufacturers of these machineries realize the the necessity of censoring the machines adequately for the specific purpose of predictive maintenance which i think it's a trend in advanced sort of manufacturing fields in robotics and machineries which are employing a lot of robotics in them there are sensors to monitor the health of the machines and parts of that but in this specific case exactly to to bridge this gap um we wanted to really connect the failure mechanics the knowledge within the failure mechanics with the predictions from machine learning and what do we mean by the failure mechanics so there is a type of analysis in the domain of engineering of machineries and structural design which is called fmea failure mode and effect analysis so what is that it is basically a procedure to utilize the research and the standards exist to analyze the moods or failures of the machine break down the moods of failures and the effect of each failure mode onto these specific type of equipment or components from bottom up and then identify the indicators that that can kind of provide a bit of symptom of these type of failure modes and then act on top of that so this is a very rigid structural engineering practice that is performed in the beginning of the design or operation and then gets updated so we were thinking um kind of the novel novelty of the the thinking that um maybe we tried to deploy was can we use this failure mode and effect analysis and build a sort of matrix that from all sort of information you get at component level um we provide a kind of more insightful picture of what can go wrong at these building blocks of the machines and then again by the use of the reliability engineering concept how we can combine these information at these building block levels and make sense from maintenance engineering point of view what is going on at least higher hierarchies like assembly levels or at system level so yeah this was the perhaps the core novelty of the approach and so the this matrix that you described is that part of the traditional way of solving like is that the input to the traditional reliability engineering approach or is that uh artifact of the solution that you've applied using deep learning new this was let's say one of the pillars of this of this approach this didn't exist so the fmea does exist this failure mode analysis the kind of cause and effect relationship mapping which you may in like data science domain use different approaches like bayesian approaches to map or build the causal mapping of the system but in our case we adopted the cause and effect maps from engineering domain and we built something called fault matrix and this fault matrix is a relational matrix a map between the failure modes and really the sensors and so in this matrix what you have is a map of what is what are the importances of each sensor in terms of redundancy and the influence in ability to detect some of the failure modes at component level so this is can be very a static matrix in terms of the importance relationship between the sun source and failure mode or it could be learned a matrix or it could be probabilistic so each let's say importance factor in terms of sensor two failure mode could be a kind of incorporating uncertainty um through the means of probably probably dense functions got it got it and so going back to the the long title that i mentioned where does deep learning come in one of the first steps of um kind of detecting the problem um is still an open question for us how we can detect accurately the problem and then link it to um you know an understandable mode for engineers but to detect it i think um traditionally at least in the field of condition-based monitoring people have been applying some fixed thresholds onto [Music] sensors and then by this means of like operating bands they were able to detect animals behavior and then based on some rules act upon those animals like sequences or so on so forth um in our case through some literature review we wanted to see can we predict the behavior of these machines um in their all possible states so these machines as you know they are operated in multiple different modes in terms of operation depends on how much load they want to apply to the machine and also because they are in operation for 30 years many several hundred thousands of hours in operation they see multiple different changes in the state of operation in the entire asset the for example in oil and gas industry the reservoir pressure and temperature depletes that the regime of operation changes so whatever sensor reading that you receive from these all all parts of the machines is subject to shift or drift or change so setting a fixed let's say set of thresholds on these first would require continuous maintenance and observation of the false alarms and so on and so forth and also it may not be fully uh kind of aware of all possible modes of operation so we were thinking we haven't seen everything of these machines operations should we use machine learning to learn from the past of these machines do we know the entire state of the data the answer was no we were in the best case we only knew 10 15 to 20 percent of the entire operational state of this complex machine sensor do you mean 10 15 of the operational state as defined by a specific set of sensors over a lifetime meaning you have the 10 percent of the lifetime of the machine or that the sensors themselves only capture some small percent of the operational state of the machine itself that's a very good question i think it's exactly the combination of both okay if if there is a machine which has been just in operation you can have some prior belief about how the sensor should behave or how we should receive the data but the rest is mystery and often identical two identical machineries have different sensor readings if you start them at the same time even set up on the same plant so this is one side of the unknown uncertainty and also a lot of operational modes as you said in terms of the state of operation is unknown throughout the life of the life span other thing is the more interesting one is the failures um the patterns of failures the trends in majority of the cases uh when you want to cover the entire let's say um 200 or 300 different individual components of these machineries you haven't seen failures of a lot of these components in their in their life so to maximize the coverage and also not make rigid assumptions you have to move away from traditional supervised learning to detect failures that doesn't work or if it works it covers maybe five percent of the problem so where did machine learning and deep learning came to help us was uh we said okay let's let's say do we know what is into our best of knowledge what is the temporal um transient healthy behavior of these machines and the sensor observations that we get we said possibly yes we have good knowledge of that to some degree and then we said okay let's let's build these models which in this case we used use recurrent neural network lstms at sensor level to monitor and learn the healthy behavior or kind of sequences of these machines so from what we get in terms of the sensor livestream data let's say if we build these representative model of the sensor from purely healthy behavior point of view can we predict lc vector into future and then checking that sort of sequence of time series versus what we actually receive we were able to predict any deviation so yes this kind of difference or residual signal for us was the outcome of using machine learning in the case of um recurring near network lstms to detect abnormalities got it got it and and so to play back at least one part of that you started talking about the traditional approach and like setting these thresholds you know like you might see in other places where you you would use anomaly detection and you know one approach might be to like automatically set the threshold but that's not what you did here what you did is create a maybe a lower level or kind of a richer model of the machine itself and didn't use a threshold based approach you you use this more residual based approach to determine when an anomaly was happening exactly so um basically this detection of abnormalities was the first sort of step let's say in this workflow as soon as you detect uh some sort of abnormal or outliers within your residual signal which is the error signal then what we could do was to use some parametric sort of techniques that you build a distribution of what is your error signal and then set some part parameter around the distribution and whatever goes out of the sort of the threshold bands you detect them as abnormalities but again we found we don't know a lot about these signals this is going to produce a lot of false alarms so what what else is out there so got into this novel approach that scientists in nasa kind of developed and applied to their telemetry data received from the space shuttles which is called non-parametric thresholding it is a kind of a dynamic nonparametric thresholding and what is this is basically trying to fit a dynamic sort of set of bands around your residual error signal and then these dynamic bands then allow you to detect any sort of abnormalities or outlier sequences um that sort of is more robust in terms of um not reacting to sort of temporal behavior of the of the sensor data that you receive has some cushioning sort of kind of buffer zone to prevent false alarms and also it is able to with the help of lstms to capture the multimodal operational phase like the heterogeneous in the in the data so the technique itself is very simple but um kind of beautiful technique it is at any given time uh define uh a wide range of threshold bands and then try to maximize the criteria by analyzing the whole historical distribution of the error and then that criteria is can we detect any sort of new set of observation that can deviate the properties of the error distribution most at any given time and that set of band is set as the new threshold dynamically so once you have this set of thresholds then you detect the strings that sits outside of the threshold and then you calculate the score for the for the sequence and that score itself is calculated based on the second moment of area of the sequence outside of the threshold got it got it so you mentioned uh in their lstms that's the way that you're modeling the state of the individual components sure yeah exactly so the use of lstms was to um exactly to what sort of algorithms are out there that can [Music] understand the temporality in the signal and the multiple modes and provide the best representation of these noisy sensor readings so in our case it was very important for us to make these predictions in in future of the healthy pattern by understanding the kind of the historical changes in the signal so long-term short-term memory type signals allowed us to kind of use the previous states of them of the system and incorporate that because of the specific architecture of these algorithms to provide the correct representation of the signal and it was very expensive to build lstms at sensor level so we talked about a few hundreds and few thousands of sensors per compressor for in our case we had to monitor about 200 sensors or 100 200 sensors per machine so building these number of lstms and optimize them based on few years of time series data with high sampling rate was a challenge one one question that i have about that is the you mentioned that you that each of the lstms is mapping to a sensor the it's kind of like you have a hierarchy right you've got these sensors these sensors presumably mapped to components and the components map to whatever the machine is i guess the reason why you do it at the sensor level is because you really don't know the relationship between the sensors and the components per se a very good point because we adapted this reliability based system which is aiming to detect the modes of failure that are active at any given time we had to um go from a to a bottom-up approach we had to utilize the entire sensors that are monitoring one component let's say in a in a compressor we have shafts we have bearings we have seal systems and impellers of the compressors so let's say in a bearing case of bearings you have bearings on either side of the shaft and you have multiple different sensors monitoring the behavior of these components a bunch of temperature sensors a bunch of flow rate that are injecting lubricants to that component a lot of vibration type sensors in multiple different directions so making sense of these of the active failure mode at any given time is a complex problem we had to understand what sort of abnormalities working together would relate to which failure modes for that we needed to know abnormalities at bottom bottom layer at sensor layer you you created the these kind of models or representations of your uh sensors and the i'm trying to put the pieces together now you've got these representations somewhere in the middle you're trying to get to the uh fault matrices or and then you've got on the other side of the fault matrices your traditional reliability engineering equations what how do you get from the lstm output to the fault matrices yes so i think the the keys lstms are providing these healthy sequences of uh healthy behavior uh as a kind of mimicking the the machine behavior from healthy point of view this is where the non-parametric came in you've got this and yes go ahead [Music] by building but creating this residual signal or error signal and then applying the parametric thresholding or non-parametric thresholding on top of that um from by comparing the healthy from actual then you calculate the score what we call anomaly score let's say at sensor level so any any sequence of time at any time stamp let's say you have an anomaly a score at sensor level so feeding an anomaly scores from 10 sensors related to one component into the fault matrix or this cause and effect sort of matrix then you get a score per component per failure mode at any given time so this is let's call it like component defective score or failure score and then having all of these failure modes lined up and components accordingly then you can have a representation of a ranking of what are the anomalies at any component level and then as you said by the use of the reliability engineering logics you can aggregate the scores at component level let's say lots of different bearings and multiple different seal systems together and aggregate the their abnormality scores together and then roll it up into the assembly level then you can aggregate again at assembly level into the system level which is compressor you can follow the same and go and cover your entire asset the by entire asset we're talking about like a gigantic oil rig or something exactly have you gotten that far we've gone up to uh critical pass machineries and crypto past machineries means those which are sitting on the critical production line you've talked a little bit about the kind of the challenge of just training all of the lstms that are involved in doing the kind of modeling here can you elaborate a little bit on what some of the the biggest challenges were well lstms are not easy to train and optimize especially that we are dealing with um quite quite a lot of data we're talking about one second sampling rate of few years of one sensor and when we talk about thousands of sensors it's quite quite extensive so lstms have multiple different hyperparameters and to get the best results you have to really invest time to optimize them the other thing was we are dealing with multiple different type of parameters we want to build these representations vibration type sensors are very different in terms of the noise patterns and temporality and conditioned dependencies historically to what has happened in order to build a more accurate representation in the future as opposed to for example temperature type signals which are they have a big delay and there is not much temporality in it the noise characteristic is different so optimizing lstms at these sensor levels was a challenge and for that we were lucky to uh collaborate with sigop which is kind of the blackbox optimization platform and then that helped us a lot using the bayesian type optimization to quickly learn the become the optimization of parameter space very quickly and efficiently because there we have the ability to optimize over multiple different objectives training time was one of our main objective as well as reducing the the loss you just said something interesting the optimization problem that you the way you ultimately frame your optimization problem wasn't solely constrained to [Music] system level criteria but you also had this was it a train time criteria or compute train time yeah exactly so i think this this is a complex sort of pipeline of connected pieces together performance of lstms in order to detect healthy patter patterns or sequences and detecting that the residual or error and feeding that into the kind of this cause and effect matrix and calculating the scores and evaluating these is entirely together is one massive optimization loop so you can look at the lstms itself in and optimize the hyper parameters there but getting the best fit there may not necessarily you know give you a global optimized path so you may get into a local optimization here so connecting all of these together building this gigantic sort of optimization loop with multiple different hyper parameters which are not necessarily all from coming from the lstm sort of high performances was was a big challenge and not so not just from the lstm but not not just from a single lstm you've got parameters from multiple lstms say 50 or 100 plus these kind of broader constraints on your runtime was your your accuracy represented as a threshold or a constraint or how did you incorporate that into your optimization the ultimate ultimate kind of objectives were into two different dimensions we had to have the human in the loop in order to evaluate the entire optimization and that was what is the intuition and engineering domain knowledge in checking whether an abnormality score at system level or assembly level is really an abnormality based on their experience and their knowledge of failure mode analysis so that was our ultimate objective that let's say we introduced a bit of supervised sort of training into the loop so we had a bit of labeling going on here in terms of what is this ultimate score at system or assembly or component level and then we had multiple hyper parameter sets at lstm stage at the scoring calculation stage and so yes we were ultimately optimizing over the frontier of reducing the false alarms um force indications and maximizing the correct detection per failure mode so that was our frontier can you elaborate a little bit on the human in the loop aspect of that um [Music] how many you know how much labeling had to go into being able to build these models that's a very good question i think we built some blueprints in hours and hours of discussions with lead rotating machineries those who have spent designing these machines for 20 years 30 years so we had lots of workshops to go through these patterns historical patterns of past 10 years or so after machine operation and then get all of these experts in agreement of what is their uh final judgment of which sort of sequences are abnormal abnormal sequences which are kind of noise artifacts which are sensor drifts and so on so forth which of them are combined together are representative of failure mode um so through that process we labeled several of these examples and then uh it was a labor intensive process for sure but to be clear what you're labeling is uh a time series data set from of what all of the sensors in a given machine or a particular sensor or i guess it couldn't it would have to be system level it has to be some level yeah so we covered um [Music] 15 or 20 critical components of these let's say turbo compressors um which are sitting on the critical path of the machine itself failure of each would result in the failure of the entire machinery downtime so we identified those and some high critical failure modes that statistically or from reliability point of view resulted in majority of the failures and then we focused on that and ultimately what the what the sig opt element of the optimization process was was kind of narrowing in on the points on the frontier that you had to care about and optimizing across that so you weren't trying to optimize across the entire state space of hundreds of lstms and these constraints and all that stuff is that the general idea that's very true yeah so hyperparameters that we're talking about are like your learning rate and the look back look back sequence which um in the case of recurring neural network is important also increase the time training time the and regularization parameters and so on so forth it was about 12 15 number of hyper parameters per per lstms to optimize and multi multi-objective basically multimetric optimization so using the bayesian optimization definitely helped a lot to quickly reduce the space of exploration and so what's what's next for this effort sure i am it's very exciting actually um so what we what we talked about today uh so far is primarily focusing on one aspect of let's say predictive maintenance domain there are multiple different things you can do if the main objective is to make the best maintenance action this is a very difficult question to answer because detecting problem is one thing making an action and a correct action is the other thing and the action space is influenced by what sort of resources do you have available which part of the geography you are operating are you in the island in an offshore 200 kilometers away from everything which supply chain complexity and the complexity of the maintenance team and the maintenance tools that you have to have on board the lead lead time of the components that you need to order and being built and tested to come may take 10 10 months or 12 months to come so you're saying that you've solved the easy part of the problem that's correct that's correct this is a very complex question to answer so what other things that can help in terms of prediction side on this first aspect before getting into prescriptive sort of uh side of maintenance is how much how much time do i have to fail till till ultimate failure this is a very valuable information so we talked about i think a lot of off-the-shelf tools that you see existing in the in the market the ge tools and so many other things are are very good in detecting the anomalies but detecting what what how much time do i have at at component level failure mode from that point of view is a very valuable because it allows them to safely operate their systems without shutdown because these machines are not designed to shut to kind of constantly shut them down to repair or do some interventions so yeah predicting time to failure is and linking that to the knowledge of failure mechanics is is an interesting topic and then and then you get to take on the what to do about it that's very true yes and i think there have been a lot of uh efforts in into research and also beautiful tools exist at the moment which are trying to optimize the action over cost over resource availabilities and over health so health sort of let's say at system level health plot or health indicator can come from these sort of approaches anomaly detection time to failure aggregation and system but understanding the cost the this is the whole set of you know new sort of formulation of what is the my cost function from operational plan point of view the the revenue that the the system is generating the criticality so and so forth to build this cost function and then resources also is also a majority of the cases is human related um and we are talking about the materials like the spare parts point of view and also human repair experts who are going to be deployed onshore or on-site or offshore kind of external supply and building a function mathematicizing that is a is a task and do you expect to be able to apply similar ideas meaning introduce machine learning or deep learning to complement existing domain knowledge to solve those other problems as well definitely definitely i think a series of like probabilistic approaches a bayesian based and bayesian networks for example or even a lot of machine learning applications in predicting different aspects of their cost function in terms of the supply chain representation of the supply chain as a kind of a sequential model and also reinforcement learning can come into effect in building the the function of your in unlearning your cost function and supply chain um also a lot of nlp people are applying in kind of feeding into sort of knowledge graphs and by building these knowledge graphs you can link the entire um asset data in terms of the operational plans what is going on in terms of you know the inner outputs and then driving information kind of from these knowledge graphs and feeding that into some sort of analytical layers to predict what is uh this consequences of actions um on top of these knowledge graphs is is another thing got it got it awesome awesome well cheyenne thanks so much for taking the time to share a bit about uh your experiences in this area very cool stuff pleasure it was a pleasure talking to you thank you thank you thanks