Marrying Physics-Based and Data-Driven ML Models with Josh Bloom - #42

Key Takeaways

[Music] hello and welcome to another episode of we'll talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam Charrington if you get my newsletter you already know this but last week we hit a major milestone for the podcast I'm excited to share that thanks to you we've served up over 500,000 plays of this show we're super grateful to everyone who's ever listening to the show send us feedback or engage with us via the site or social media thanks also to all of our gifts especially those who started out as listeners and later became guests like Evan Wright and Sarah throw we've come a long way in a short amount of time and we couldn't have done it without you next up we're ready to announce the first winner of our AI conference giveaway drum roll please congratulations to shinu from New Haven Connecticut shinu was one of only 5 people to complete every possible method of entry and clearly it paid off shinu we look forward to seeing you in San Francisco we're still working to finalize our second winner so stay tuned to our Twitter feed at at twilly I for updates if you didn't win the contest would still want to join us at the artificial intelligence conference in San Francisco head over to tamil AI com slash go AI SF and enter code PC twimble for 20% of the cost of most packages next we're less than a week away from the first ever meeting of our online paper reading group the twimble online Meetup our first discussion will be on the recent paper from Apple learning from simulated and unsupervised images through adversarial training if you haven't registered yet head over to Twilio comm slash Meetup and you can do so there the meetup will be Wednesday August 16th at 11 a.m. Pacific time it will be recorded for those who can't participate live if you've already registered but haven't read the paper yet now's the time to get started and if you have started reading and have questions please post them over on the Meetup slack channel which you should have been invited to after registering before we get to the main event I'd like to give a quick shout out to our friends over at Banzai for their continued support of the podcast and our industrial AI series Banzai offers an AI platform that lets enterprises build and deploy intelligent systems for industrial applications if you haven't investigated the company in their platform before I think you'll find it interesting you can find more information about them in their early access program at bonds AI slash twill Malay I finally about today's show I recently had a chance to catch up with a friend and friend of the show Josh Blum vice president of data and analytics a GE digital if you've been listening for a while you know that Josh was on the show around this time last year just prior to the acquisition of his company wise IO by GE Digital it was great to catch up with Josh on his journey within GE and the work his team is doing around industrial AI now that they're part of one of the world's biggest industrial companies we talked about some really interesting things in the show including how his team is using autoencoders to create training data sets and how they incorporate knowledge of physics and physical systems within their machine learning models of course why is that io @ GE digital is a sponsor of my industrial AI research and this podcast series and for that we're extremely grateful to learn more about wise visit their site at wise IO to learn more about wise io @ GE digital visit their site at wise io all right everyone I am on the line with Joshua bloom josh is VP of data and analytics at GE digital and if Josh's name sounds familiar it's because it should he was our fifth guest here on the show and that was back in September of 2016 Josh welcome back to the show you're our first repeat guest awesome thanks for having me I'm super excited to have you on the show and I encourage folks who who haven't already listened to that so or even haven't listened to it recently to go back and listen to it again it was 2005 and it's been one of our most popular shows of all time so you can really really dig deep into Josh's background by going back and listening through that show but for now just why don't you give us a little bit of your background and catch us up on what you've been up to recently yeah so I started off in physics and astrophysics and did a PhD at Caltech went to Harvard as a postdoc and then got really interested in robot eyes Asian and I was then that I started getting excited about machine learning as I realized that we had some big data problems coming down the pike and when we needed to do discovery on new images let's say coming from telescopes at that point my colleagues were basically saying I would just hire more grad students to look at the data so that's how I came upon machine learning a little over 10 years ago fast-forward a bit we went up building up a kind of end-to-end system to do some astronomy projects with data and machine learning and then with the research group wound up starting a company called wise io based out of Berkeley California over time we wound up building out a set of products in customer support integrating with Zendesk and Salesforce to help support agents basically become more efficient and delight their customers even more last year I think around the time that we had our first conversation Sam we had some brought interest from a number of different companies and GE Digital wound up becoming very interesting and compelling for us and I'm sure we'll get into that but we announced our acquisition by G digital in November of last year in 2016 and since then have been working within GE digital for a much broader GE ecosystem and Gigi's customers and certainly happy to talk to you about what we've been doing and you know how we've made the transition from really customer facing and consumer internet to industrial Internet great that's exactly what I want to focus on for this show and in fact we can start out by me thanking you n GE for graciously sponsoring our industrial AI both the research and the podcast well it's our pleasure and needless to say it's been fun for me personally and for those around me at wives within G digital to also have you go through the same sort of process as we wind up learning about this you know industrial machine learning world in some sense we've lived sort of parallel tracks are coming to realize you know how important it is what kind of value there is and how different it is from the other types of machine learning and AI that's done still in industry but not in the sort of hardcore machine industrial context so maybe let's start there with the kinds of things you've been learning you are in you know I can't think of a more target-rich environment the way I think you've once explained your role at GE is to kind of AI all the things or at least be a part of that and there are certainly a lot of things there that a lot of industrial things there to be a I'd if you will have we just verbs AI I think you get the credit for coining that if that's right how's that been going and and what have you been learning yeah I mean I some sense the practical nature of what it means to do machine learning in production a scale with fault tolerance is the same sort of thing that we took from our his work in previous products set and applied it to the sorts of problems that GE has in front of it and I'd say the most practical thing is to say that we shouldn't be a Iying all the things that a lot of things don't need AI and oftentimes when I give talks internally within GE one of the big things that I'll challenge people with is this notion that everything needs you know machine learning or if all we did is just apply tensor flow to it then all our problems would be solved we don't believe that I think that people have been working in this field for a long time understand that very deeply and so part of our mission is to help people within GE and eventually Gigi's customers to understand what it what what are the workflows what's the type of data where you know advanced analytics or machine learning or even more broadly AI can be used to affect better business outcomes and it's kind of with that lens of why are you doing this that we're able to say yes to a bunch of things but say no to a bunch of other types of projects where you know traditional business rules may just make great sense or these are problems that are massively complicated these are machines that have very good physical models that describe future behavior based on past behavior and that's good enough given the business outcome but I'd say one of the things that's changed a lot for us is to understand how important this is not just at the scale not just in the details of what it means to get a right or wrong answer and you know for your for your listeners to understand that you know you're always trying to optimize something when it comes to accuracy you know an area under a curve or some false positive rate at a fixed false negative rate but as we start imagining and we'll get I think into some of the details in this interview about some of the specific projects you can imagine that there are new places on the on the ROC curve where you don't ever want to be wrong right where you always need to be right and this changes the nature of how you do data science it's changing the nature of how you build products around it but I'd say the thing that we've learned is that that view of not everything needs machine learning to get a pet a great business outcome but everything needs to work within the current context of not just you know an old industrial company like GE which has been around for 125 years but with the business processes that have been built up in some of the you know oldest most analog parts of you know of the economy let alone not being digital not let alone not being you know software savvy trying to bring products into a place that has sort of used to doing things with the status quo because they just work is a challenge in and of itself and all of that in some sense makes great sense when you understand that what we're talking about machines that affect our lives the way one of the distinctions I make between the consumer Internet of Things and the industrial Internet of Things is that you know when your Fitbit breaks you call up customer support and you sort of complain but when you're jet engine has a problem that can have some serious consequences and so the stakes are a lot higher and because of that you know there's a whole regulatory environment which is something that very few people in the consumer internet have had to deal with yes you sort of have to think about PII you have to think about HIPAA compliance and things like that but now if you're talking about you know a regulatory oversight body with the FAA or the FDA there is some extra boundary conditions that's put upon us as we start thinking about bringing machine learning into those those sorts of worlds your team isn't really targeting you know or charted with kind of making sweeping revolutionary changes at GE but rather you know you're proudly taking a much more incremental approach you have to start somewhere and again taking a very practical view of what it means to transform a existing workflow that involves data that involves people that involves you know physics based models that involves decision rules and then start bringing that into machine learning centric workflow there's a lot of different stakeholders involved and many times people have already tried bringing machine learning in and failed for various different reasons and so one of the sharp elbows that we wound up building up as an independent company and we and we bring to GE now is around that notion of you know what are the problems that you should be solving and in particular should you be going after the you know highest value most complex ones or does it should you really just start somewhere and we really think about low-hanging fruit and in some sense that's our lens is within the industrial context where is the low-hanging fruit where it's so obvious that AI can have a measurable impact not just on you know things like accuracy or you know time to make a prediction or something like that but in real dollar terms and the way I like to talk about it is we're not trying to solve the problems that end with a be they shouldn't end with a K and so we're sort of in that hand world where you know at the millions of dollars a year level if we have impact there and we can start working with the individual business units and their customers to start helping people understand even how to structure a new problem around AI and understand what it means to do data governance right what if you know how you wind up basically building up a an AI first product from scratch then we wound up when we went up winning because we get to multiplex across multiple you know sort of internal and external customers one of the things that I specifically remember about this conversation we had on this point was you talked about kind of impact of 1% in your world yeah that's that's that's right I mean that really you know in some sense gets to the scale that yes you know if you have a 1% improvement you know in a in a product and a workflow that is making you know hundreds of K a year and revenue that's not a really big deal but if you do have a 1% improvement let's say an efficiency in you know a billion dollar product that starts to get to be the real dollar so in some sense you know what I just said before of taking the low-hanging fruit and not trying to solve the really big problems we get away with within the context of GE just because the scale at which we're talking is just so immense to just give you a sense of it for instance when a jet engine finishes a flight you know call it a five-hour flight on average there's a bet a terabyte of data that's generated just from that one engine and you can imagine even the process of offloading that data from the airplane after it's landed and then getting it into even a data like that itself can be pretty complicated but then doing you know sort of real-time analytics on that and making some decisions from a preventative maintenance perspective is you know one of the really important things that we have to be able to do but now if you think about well there's 50,000 jet engines that are flying every day that gives you just some sense of the enormity of the scale right so each each flight is basically a day of Twitter data and then you go you know factor of a few orders of magnitude larger than that so for us you know yes we get to work on you know the quote unquote low-hanging fruit with fairly large dollar numbers in part because making small incremental improvements in the workflows that involve lots of data for very expensive important machines is just sort of the reality of where the industrial internet is right now so can we talk a little bit about some of the use cases that you've seen are there ones that you can walk us through yeah so I can't go into you know all the specific details but the one that I just spoke about within aviation is an important one and there's something that's been discussed publicly is the need to have advanced analytics applies to data that's coming off of airplane engines to achieve better outcomes and one of the things that is important to recognize about many of these industrial use cases is there's a huge value to being able to understand ahead of time when something is going to break or whether something is in trouble and you know that is where you get into some very interesting data science problems of especially given a lot of these these objects very rarely fail is how you build up you know sort of counterfactual evidence so that you can test your models offline I mean the easiest thing to do would be to build the machine learning model that says take every wing every engine off the wing after every flight and you know by golly you'd find every single problem because somebody would it would take it off but then you know that whole industry would come to a halt we would probably destroy the world's economy just given the extra latency juice of what it would take to retire an airplane every single day right so that obviously doesn't make sense so there you know our false negatives would be basically zero but our false positives would be just uncomfortably high the other approach is to say you know everything's working all the time and you know for the most part you'd be right and the number that I have in my head is that the sort of failure modes are only sort of you know a few in a million flights will there be a significant problem with an engine which is why we have multiple engines on a plane and so you're in very very small number statistics land and you can't really ever know if I said take this engine off the wing whether it would have failed had I not said that there may be some diagnostic evidence you could see when you actually look at it but you can't ever gather the counterfactual of what if I didn't do this and I can't really run a B test either where you say well I think you should take these up the wing but I'm not going to say anything about that that obviously has its own problems as well right so you're in this very interesting dance where it's lots of data yet in some sense it's a small data problem because you only get the sort of bad or rare anomaly events every now and then and even when you get those you wind up having yep they're all sort of Anna Karenina like they're all like you know unhappy families right they're all different in their own way you know that's a real challenge from the data science perspective that's where some of the interesting innovation has to happen from an R&D perspective is to work in this you know kind of really long tail world so that's that's kind of one family of use cases that were interested in but imagine before we kind of brought into other use cases you know I'm sure people are asking like okay how do you address that technically from a data science perspective what are some of the the ways that you tackle that problem well in some sense it comes back down to do we even tackle that specific one or do we tackle ones that are adjacent to that and getting back to some of the work that we did in customer support and I think is one of the really core design principles of how you think about building machine learning products that is building assistive tools that wind up not sort of making a decision by themselves but actually provide information and insights to analysts who are looking at the data and there are analysts who are looking as the results of data coming off of all these engines so instead of saying yes this is going to fail and take it off or no it's not there's an adjacent problem where you can wind up saying you know I'll create a ranked prioritized list of the engines that I think in analysts might want to look at and then you let the domain experts in that world then go through the path and make some decisions on that so it becomes kind of an accelerant and a kind of an efficiency pray rather than a you know black or white kind of you know almost trakone ian type of thing so we're not anywhere near the point where machines are going to wind up generating you know work orders without any people in the loop so when you wind up pulling it back a little bit from the this is going to fail and this is okay too I think this is something somebody might want to look at it turns out that analysts look at lots of things right and so they're often digging in to a specific engine to understand you know what's happening with it and so we have a lot of that data from from the past now it's not a very long tail problem we make it sort of more evenly balanced of you know X percent look like they're fine just from the very high-level overview you know 1 minus X percent it looks like they need to get some more more work done or somebody needs to dig into the data a little bit more and as long as X you know is close to 0.5 or even if it's 0.1 you're in pretty good shape because then you can apply sort of classic supervised machine learning problems to that so that that's sort of one trick that we have is to take something out of the you know very sort of rare regime and try to bring it back into a world where it still has value you could still measure that value but it becomes kind of an empowerment tool rather than something that's making absolute decisions that's kind of one thing that we've brought from our past lives into the into this one I guess the other one is you know the nature of the data is very different you've got data coming from lots of different subsystems tends to be time series data in the past we had worked a lot with natural language processing and you know one thing is very powerful with this sort of data is that if you have a lot of it there are techniques one can view let's say within the deep learning world where you can in a very unsupervised way you know build up some capability of generating features out of that and then do anomaly detection off of those features or just do direct classification off of those features and so you get to leverage lots of the you know quote unquote normal data and normal behavior and then use that to be able to make inference on the things that look like they're out of band so you know for us being able to apply some of the you know cutting-edge techniques in let's say recurrent neural Nets and unsupervised learning around these time series data sets is very interesting and I guess the third part of that which is coupled to the other two is trying to do this all at scale and trying to do this all and as real-time as needed for the specific problem and we've really crossed over in terms of our back end in front of when in terms of what's needed from sort of large single machines in a multi-core environment to you know a multi node type of environment so doing deep learning across multiple GPU instances is something that our infrastructure that we had built before has got to adapt to what I heard was you talked about using deep learning to to generate features that would allow you to train more traditional supervised models and that actually that reminds me of there's a cell you may not be aware of this but we're starting a paper reading group associated with the podcast and the first version that this meetup is going to be on the 16th of August and the paper that we're going to go through is one of the papers from cvpr we're a team at Apple basically used the generative adversarial network to generate data to then train a supervised model - I think supervised right more actually I'm not clear on this because I haven't read the paper yet but I will buy it just by the sixteenth but to set the point that I wanted to pick out is I've heard this notion a couple of times around using deep neural networks to generate training data for you know either supervised or more traditional models a few times and I wanted to make sure that that's what I heard you say you were doing and also kind of get some feedback from you on how broadly applicable are you seeing that across the various use cases you're looking at are you doing a lot of that okay so to be clear we're not using Ganz to generate training data there's another approach that's it's called autoencoders that allow you to generate features in an unsupervised sense so it's adjacent but it's quite a different problem what I will say though in general about ganz and there's only you know kind of a few papers out on this in this context is that it's actually kind of interesting if you think about the data privacy and the sensitivity around some of the data that we have access to at GE you know passing data around and clearly it needs to be done in it you know highly cover and highly regulatory approved way isn't isn't always the best thing and especially when it's when it's very large so you can imagine that there are use cases we're different groups within you know within the same company may need to get access to data but instead of sending the data itself over you can imagine building a scan that's able to generate data that's like the original data so if you have very sensitive data that you don't want moving outside of your walls garden or your data lake you can imagine building again that essentially simulates that and instead of handing somebody you know the keys of the original data you could have them you know keys began which you know if we're right of a reason fell into you know into hands that were in supposed to see it you know could provably not be able to reproduce the original data yes folks who got access to this band would in principle be able to build you know machine learning models against that and so I think it's a very interesting and clever way to start thinking about passing you know information about a set of data around without actually having to pass that data around and so being able to build models that learn from you know different groups and their data without those groups having to share data amongst each other or without having to aggregate it all into one physical place is of great interest to us and it's not just sort of an interesting thing to do in many cases it's a necessity if we want to build great models and we can't you know even see the data or we can't Fedder a tit into a single data link we have to have you know really clear paths to being able to do that and again there's academic literature on this but there's not a whole lot of work that's being done on this in practice that's kind of one interesting regime so you bringing this up as I think an important one and the other one that we touched on before at the top was around sort of the marriage of physics-based and data driven models you know unlike again in the consumer internet where you know you build a data driven model around customer behavior or around you know actions or on sentiment cetera you can try to build some sort of latent understanding about how the brain works etc but there's a very complex you know biological systems effectively that are giving rise to the data that you wind up trying to apply on there is no physics behind you no recommendation engines there's no sort of core principles there yet in the industrial world you've got jet engines you've got MRI machines you've got wind turbines nuclear power plants and these are all built up by physical objects that if you knew all the physics of them then you wouldn't need any data because you'd be able to predict exactly what's going to happen in the future so for again a preventative maintenance perspective you'd be just fine but as we all know even in very simple physical systems we often don't know all the physics Umbro while GE has I think some of the world's experts in all the very various different sub domains in material science etc building up complex physics models you know I think of it as physics models only getting us you know 90% of the way there to a great answer and then adding is sort of data-driven you know layer on top of that is the path that we're we're seeking so rather than what we did in the past where you just take you know effectively a fully data-driven model to get your outcomes we're quite interested in understanding you know how in a rigorous way do we combine the outputs of physical models essentially as the inputs to data-driven ones in addition to all the sensor data that you're getting so I'd say that's a really critical distinction it's also a huge amount of white space that we see in the industrial machine learning world and that's something that GES been pursuing or evangelizing for a while through this notion of digital twin can you talk a little bit about that and the role that it plays in the work you're doing around ml nai yes a digital twin for those that hasn't heard the term is a idea and an implementation of an idea that every physical object should have a virtual version of it that you know could live in the cloud or if it's very sensitive can live in an on-premise environment and that digital version should be kept up to date with the physical version of it and it should know about its maintenance history it should know in the context of in an asset model it should know you know if this is a part in a large machine you know it should know about the machine itself so it's a very base layer I think of a digital screen as a digital representation of a physical asset and all the data that's available about it both historically and then in real time where AI and advanced analytics comes in on top of that is to say well given all of this data you know can I make a predictive statement about what's going to happen to the physical object by interrogating the digital version of that so rather than having to you know ping a hard drive which is you know on the device itself and try to pull out data we need strategies that take data from the from those edge devices bring them into the cloud and then it allows me you know in a more relaxed cloud environment to be able to ask questions of that maybe make some maybe take some actions based on it and then then the next step after that of course is to take the results of some of those predictions and push them back into the physical device itself and potentially even update things like you know configuration variables based on predicted outcomes you know I don't think we're really there yet across a wide swath of GE assets but that my sense is where a huge amount of value winds up coming in if you're able to build you know machine learning models now not just against this one twin but against you know all the twins in the same asset class and use those you know models not just on one customers data but across all customers data to be able to you know get better outcomes hmm and so this is somewhat related to the role of simulation in building mln AI systems for industrial applications in general right we talked previously in a conversation about how you can't just take the engine off and you know put it through its paces to generate data sets you know what have you learned about the process of using simulation as a way to create these models yeah so to be honest I haven't learned that much doesn't mean I can't anticipate on it I can't say that I've learned a tremendous amount just that those aren't the sorts of problems that we've been directly are exposing ourselves to clearly there's a you know kind of reinforcement learning play in that conversation about being able to simulate the environment or the results of an action that you wind up taking and being able to you know build models offline before you wind up deploying it into into the field we haven't ourselves been working on sort of that kind of robotics angle but that's obviously really important that said you know simulations get back to that the physics based model that I was describing earlier in some sense I think of physics based models as essentially simulation no you've got you know a simulation of your physical object because you think you understand most the physics based on the whole history of what's happened to that object if you've done you're going with simulation right you should have some you know bands you know uncertainty bands into what state is happening next then again could you build a you know machine learning model that's effectively taking the results of that simulation and then using that as your you know more or less your physics plus data-driven model yes we've got you know I think a sort of simpler notion of how you do that which is just taking the base predictions out from the physics based model and use that as inputs in addition to all the sensor data you know to build a machine learning model off of that can you elaborate on what you mean by that so yeah let's let me we can take an example you know let's say that you've got a wind turbine and you've got a prediction of what the winds is going to be in an hour from now and let's say that there are configuration variables that one might want to set effectively in real time based on you know essentially with it with the optimization goal of maximizing the the energy output so you know we think about it as rotating around to try to get the optimal wind direction so now based on you know some data that's coming in and predictions about the future based on what we know about the physics of the object that's going to be spinning and how long it takes for us to spin you can imagine you know given a set of inputs like what the weather is going to be and what the weather was an hour ago and how fast the turbine spinning now you could run it through effectively a physics simulation that says you know if I turn at this amount I'm expecting to get this this energy yield out if I turn it by this amount I'm expecting to get this energy yield out what I would posit is that one can take the results of those predictions you know think about it and very simply as you know an efficiency curve as a function of you know as muta Langille of the rotation of the blade you know there's going to be a place where that's optimized and right clearly that number is going to be wrong because there will be other physics of that object let's say that objects get a little kink in it or it's at a slight tilt or the the models of the of the winds are always systematically off by you know five degrees in the orientation what I would then do is say well in the past we've had all these turbines you know choosing a next best action for itself and those haven't been necessarily optimal how could we take all of the data that we have coming in and build a model off of that to try to get a more optimal answer and of course what you would do in some sense because you have multiple turbines in the field is try different potential outputs based on what's actually what the model actually predicts and then as you get the results back you say well that looks pretty good that's how you're effectively building up a continually learning model you could call it a reinforced model that you could then deploy and get better and better over time so you use again the predictions from you know the non machine learning part of your model and use that as an input to the machine learning model that's a pretty fascinating take at reinforcement learning right we think about reinforcement learning is you've got these physical systems perhaps that maybe you know maybe super simplistic like an Atari game or maybe you know a simulation of a robotic system that has you know some degree of fidelity to real life and you're using the simulation environment as a way to give you feedback on you know what you know what happens in real life and so the model is kind of acting in this simulation environment what you're describing is kind of flipping that on its head and making you know real life your wind turbine farm your simulation environment in a set innocent I guess not in the sense of simulation but in the sense of the environment in which your models take you know control deliberate actions to try to minimize you know minimize the error maximize efficiency yeah and to be clear I don't I don't think that's a unique view that I have you know one of I say the highest value results as I've seen come out of google's deepmind is an optimization on you know HVAC usage in large you know compute environments and they're you know you can imagine you've got 17 different levers to push up and down and you know there is no a priori understanding other than the physical thermodynamic physics of how our room responds to HVAC other than to say that you've got a whole bunch of data coming in like such well what's the server load on every single object where is it located in my data center and all I can do is move you know these 17 levers up and down you know that's something that your simulated environment is actually the real environment and getting to turn those levers up and down is something that you wind up learning how to do over time because you've got a very clear optimization metric which is you know how do I decrease my energy costs of pumping AC into this into this room while still maintaining a level of reliability on the on the machines to not go over there you know expected heat loads so you know I think that's a very clear example in an industrial context in some sense where that sort of notion of reinforcement learning winds up flying out to be clear though I would say that reinforcement learning as as you've described as I was describing in the industrial context is really kind of one end of the spectrum of you know what we'll call continual learning and you know the sort of other end of the spectrum is you build a model on static data you deploy it into production and you grab feedback of whether you're right or wrong and then you know a year later you build another model based on that feedback and you deploy it that's sort of a very sort of gradual continual learning right punctuated and then step farther forward into that sort of what we were doing in wise when we were doing customer support where you wind up having a model that's rebuild everyday because the world of customer support is changing fairly rapidly and those are deployed and it's taking all the feedback of what you've just learned over the last day and then you can imagine another one where you know it's sort of a cyber security environment where you want to have a model which is updating and itself based on different threats that are coming in that could be an update on a minute timescale the kind of continual true continuous like real-time learning where you have these online models that are just getting better and better over time and adapting to changing environment is you know a very natural place to be and so I see that all as part of a continuum now has vast implications about the engineering behind it and even the data science and the certain techniques you would use but conceptually I think it's all sort of very similar I think the distinction that I thought I heard there and we can go back to the deep mind HVAC example in the context of this continuous learning spectrum that you outlined is you know they're you know clearly they're modeling a physical system they're deploying models out to a physical system they are you know continually optimizing the you know this model and you know getting to a system that has you know getting to a model that can control their seventeen lovers in a way that you know produces optimal output or at least way better to use a technical term output over a given period of time what I thought I heard you describing and the wind turbine example was if we kind of map that to this continual learning spectrum and say that you know their time scale is you know their feedback loop is operating at such a scale that it's you know near continual what I thought I heard you describing was almost like accelerated continual learning meaning we take this model and then you know we push it out to the physical devices again in this case the wind turbines and direct them to act in specific ways you know not to pursue the the plan that is outlined in this model but to you know deviate from that in a way that we think will accelerate learning and thus produce a better model more quickly dead did I make all that up word were you saying something like that no I think I think you've got it in the missing piece of why I see that connected to the HVAC example is that in principle and I don't know this to be true or not you also have a thermodynamic model of what what would happen if you you know through lever for up higher and increase the you know the HVAC in that region of the room you in principle could model that and you can imagine that instead of just using the data coming off of the individual computers as the input instead you could also use that data plus a thermodynamic prediction about what would happen if you made a change and I agree this could actually be an accelerant because in a world again where you know all the physics you wouldn't need a domain driven model you would just are data driven model you would just take all the heat loads and you just crunch some big supercomputer which itself could add to heat load in the room but then you'd wind up being able to say very precisely if I change all the levers in every single possible combination what is the optimal output but that's obviously a very hard on this intractable problem given the complexity of even of even a data server room so I connect those two examples in part because there are physical systems and in both cases you have the potential whether you use it or not using you know the you know true thermodynamics and physical modeling of what the expected output should be and instead of you know having to explore that space in a purely data-driven way you then have the ability to explore it in a sort of simulated way and at least do an exploration and you're in the physic in the real world around where you think the good answer is going to wind up being is there anything that you've learned or any direction that you can point us in terms of the you know the very practical tactical approaches to integrating the you know the physics into the modeling process and the models themselves yeah I mean you know in a time-series sense you know if you've got a physical system that is you know behaving effectively like a sine wave and there's a you know call it a linear oscillator that's involved in producing the data you can imagine you know fitting with your physical model so this is if you haven't prepped parametrized the data that you have to your physical model and getting out of in ternal parameters that you know more or less gives you a good measurement of path and you can then you can then use to make a prediction about the future so again if it's a sinusoidal model you'd fit you know the spring constant and the mass of the object just to make it really simple and then you get you know in the if it's a perfect model and a particular surprise the problem you'd get basically just a set of residuals from your fit that's consistent with the noise properties of the data but oftentimes in more complex systems you might have some residuals that are you know correlated in time and not zero and not consistent with the errors which means that there's more things going on that you know so instead of building a machine learning model on this you know large sinusoidal wave why not just build a machine learning model on the residuals of the data and there you could then bring in other data points you can bring in metadata it becomes very powerful in some sense we're move away you know the signal that you know about and only model the signal that you know is unmodeled so you know if I've only got a certain number of data points and I've only got a finite size model if I don't imbue any physical understanding of this into this into what I've got you know I basically now have to fit a sinusoid using my machine learning while they can do that of course but then you're you're using your your power up in something that's knowable from other by other means but imagine you measured that right and you said well I know it's a mass in the spring and I get these measurements but boy you know I can't predict the next time step why is that so you you know do what I just said you subtract off essentially your physical model and then what you wind up realizing is the residuals are growing in time it's because you forgot to include friction well now your domain-driven model is going to basically learn what the you know friction constant is so that it winds up getting a better prediction when you combine both of those two together and it may have had a harder time finding that if you just said I don't know anything about the system let's just use pure data to figure it out so I think that the whole point here is that these are physical systems that have the potential to be modelled and yet that our modeling capability on the physics side is is imperfect because we don't know all the physics yet that's clearly in some sense prior information that we should be using and then you know removing that out of the original signal and then only trying to predict what those residuals are so that we get a better answer you know when you talk to or when I've talked to help try to be more precise here some of the folks from the deep learning perspective you know they kind of say to probably poorly paraphrase them in a way that they disagree with forget about all the physical model stuff like what's cool about this deep learning stuff is that it'll figure everything out right so why worry about trying to incorporate these models you know let's just throw times and tons and tons of data at this thing and you know the network will figure it out and that's always been counterintuitive to me and so you know I just wanted to kind of poke you at this a little bit to make sure really clear and you know us as a community are really clear on why at least in this domain that the physical models are important and can be very powerful well looks at it to put ourselves in the minds of the people that made those sorts of statements there's great evidence that they're correct that you have a whole history decades wrong computer vision where you know people are trying to come up with essentially physical models of you know what it is that a machine is seeing and building you know a very deep understanding based on our understanding of the physics of a vision into being able to make predictions being able to do segmentation being able to make classifications and then you know deep learning matures the large large data sets benchmark data sets went up coming out and all of a sudden all the old models and all the old ways just fall by the wayside so there's example in the computer vision world there's even animals of Minnesota languages yeah it's going to say in the NRP world you know famous NLP person I think from the 70s you know that every time I fire a linguist my model improves right this crazy notion now that yeah why do I need to have a you know complex understanding of how language works when in the end all I really need to be able to do is just throw massive amounts of data at a network that's capable of learning it so there's certainly examples where you know physical modeling or you know theoretical theoretically hierarchical models of how language works just basically you know we're inferior once you had enough data and you had you know sophisticated enough networks but the operative word that you said or phrases tons of data and that's problem relative that's problem well exactly that's problem relative again let's come back to the jet engine example yeah we've got tons of data and more data and jet engine world and principle than you know just in you know any data lake of any you know computer vision researcher so that then you would say well we've got more data so you just throw it at and throw at it except in this case as I was saying before we have so little examples of things going wrong because these engines are so good and so robust that you have to appeal to physics in some sense you have to look at a physical understanding of these objects and the different failure modes because you know in computer simulation land or in just physical simulation land more broadly you can test a whole bunch of different things that never get tested or seen in the real world and so you can build off a whole you know a whole bunch of failure mode environments that you know if you start seeing something like that happen in real data that becomes a trigger point and you say look we've got a problem that's upcoming let's let's take care of it whereas it that purely data-driven model my hunch is at best it could say this is something we haven't seen before but it can't tell you what's going to happen in the end because you don't have a predictive models literally never happened before in any of the data you've ever collected yet it is something that you could fit you know a physical model to and show well given all the data what's happening last you know 10 days this outcome is now expected and again you won't be exactly right on those but I would argue that and those are great examples your physics based models are going to wind up trumping you know purely ADA driven models and in the end I think it's going to become clear that it's going to be the combination of both of those notions that will make the most powerful most robust outcomes hmm great well this has been an awesome follow-up discussion and I'm super excited to have you as our first repeat guest back on the podcast is there anything else that you'd like to leave the our listeners with well first of all I would love to be the first three piece deck as well so we can look forward to that in the future and maybe somebody has a model for who that may may happen but yeah what I will say is it is a very interesting time to be crossing over from the consumer Internet working on you know valued problems for you know people and their interactions I think we're working on at GE digital you know high-value problems for people living their lives ready to get on airplanes your house is powered by a power plant somewhere etc you go to a doctor and generally GE machines or the things taking pictures of you and your insights so your life as a person depends upon the the industrial Internet of Things and you know it's a great time to be part of that and there's a massive amount of work to be done cuz I mean you're hiring oh yes you're hiring please I would love to have any of your readers contact me directly emails easy it's just Josh bloom at chicom or you can tweet at me I'm just prof PR ofj SB and love to love to hear from you I think the important other thing is it's not just you know are you a machine learning expert in this one little you know realm of time-series multi special time series for blah blah blah it's we're really looking for people that you know just know how to scale computation and work with data under very no restrictive environments around security and governance so for me it's it's exciting not just you know thinking about it from the m/l perspective but from the engineering perspective fantastic well thank you so much sauce it's great to great to catch up great to catch up with you as well thanks so much for having me on and love the series and love what you've been doing thank you [Music] all right everyone that's our show for today thanks so much for listening and for your continued feedback and support for the notes for this episode to ask any questions or to let us know how you like the show leave a comment on the show notes page at twill Malaya comm slash talks +42 thanks again to our sponsors bonsai and yzo @ge digital for more information about bonsai visit bon ji / - mo AI and for more on wise visit wise io don't forget to register for our upcoming online meet up at twill Malaya comm / meetup and my newsletter at swim le i com / newsletter thanks again for listening and catch you next time