Fireside chat: Emergent innovation

Key Takeaways

hello everybody my name is janae flager i'm the chief medical scientist for microsoft research based up out of our cambridge lab and this is my sincere pleasure to host nubar atheian to discuss the topic of emerging innovation at the research summit this year nubar is a distinguished entrepreneur life science leader humanitarian and somebody i've personally looked up to for a number of years and it's my sincere pleasure and my thanks to nubar and his team for for joining us uh this afternoon uh newborn today i think we'll be exploring a number of areas and learn about the experiences that have shaped you your mission to improve the human condition through science and my say thanks again to you for for taking the time to join us personally i find your pers your story absolutely fascinating and your personal story seems to be something that's completely central to your approach around innovation i wonder if you could share with this audience a little bit of a story about yourself and what's led you into the health industry well first generate thanks for the kind words and introduction and uh it's a pleasure to be able to have this uh conversation with you today um you know i think as as in many people i'm sure you end up doing what you're doing and then you realize that there were things that moved you in that direction although you're less conscious about them uh as that's happening and so it's only after a period of time that some of the moving kind of forces appear and so i think in my case i do believe at this point that the degree to which i tend to be comfortable being uncomfortable and being outright afraid um the degree to which i'm less accepting of conformity and and kind of um conventional wisdom and am willing to risk uh you know sounding dumb or unreasonable in order to make new discoveries all of those things i think are rooted in somewhat the past that i've lived you know i i grew up uh in lebanon in the middle east uh basically saw a civil war for a portion of of of my life um escaped and re you know my father my mother rerouted us in montreal canada i'd never seen snow before let alone you know kind of the culture and without really knowing it as a teenager kind of got accustomed to being you know rootless essentially and you might say well that can't be a good thing but it turns out that if you're going to live a life of innovation or i'd call it extreme innovation in a funny way roots are not that useful because if you want to keep planting your roots in the future among the things that that prevents that is having deeply planted roots in the past and by dent of my personal history whether it's in my lifetime or even the my family's history which for generations has been unfortunately in various ways chased out of where we where we lived uh probably the disadvantage that that represented ended up being an advantage to what i do today because i'm less looking for the kind of stability you get by doing things you're supposed to know or doing things you've done before uh so yeah that those are the thoughts at least that over the past few years i've come to think about the previous 30 years of my journey amazing and if you take all of that you've done engineering you've done biotech and now you you're the head of an organization called flagship pioneering i'd love perhaps you could share with us taking those experiences what led you into healthcare in particular and could you tell us also a little bit about flagship pioneering well sure so so i ended up entering kind of the field of biology as a graduate student i had an undergraduate degree from from mcgill university in in montreal where i grew up ultimately in chemical engineering and chemical engineers in the prior century had really been the basis for the petrochemical industry and many of its derivative products and and generally dealt with chemistry and in the early 80s when i when i studied there the kind of only new cutting edge thing uh that was happening was the beginnings of what became known as biotechnology and and and of course this was a scientific endeavor in the first place but since people wanted to make products the closest thing to biological products were chemical products or so we thought and so the industry started using the technologies of producing chemicals at large scale to try to make proteins which of course was a ill-fated attempt because these are quite different materials and quite different properties but nobody knew that so i ended up getting drawn to that challenge again being a little bit out of the the comfort zone of a chemical engineer and i came to mit to do a phd in in what was a brand new program called biochemical engineering i was the first graduate of that of that center at mit in 1987 and that planted me firmly in whatever bio and chemical engineers would do um and and but but interestingly that what got me into healthcare uh beyond just that is that i was less interested in making large quantities of things than understanding the molecular mechanism of these machines you know most things that an engineer engineers are man-made and there is a portion of engineering which is usually called reverse engineering which is other people reverse engineering what man-made but in nature you don't have that man-made presumption and yet you need to reverse engineer so you can understand the mechanism of this you know complex multiple interacting parts machine and so i was drawn to that my phd involved work in that area and i've in one way pursued that for the following 35 years in one way or the other either by making better technologies and tools by which to do it or to use the then newly understood mechanisms to make better drugs or better uh even agricultural products or nutritional products all places where this kind of nature derived complexity either guides uh the the designs that you then use or the something that you need to fix still having understood what's going on it's almost a beautiful segue it seems to some of the processes that have led you to then think about the principles behind flagship it seems i wonder if you could tell our audience a little bit around flagship pioneering i wonder how many of them are aware of flagship i i'm almost certain everybody will know of moderna but will they know about the link between madonna and flagship and perhaps perhaps you could share that sure where you know well there's a there's a fractal nature to to this kind of kind of organization in the sense that every bit as much as everybody knows about covet vaccines but nobody knew about moderna everybody now knows about moderna but they don't know about flagship and of course that's because you know and it's it's all about kind of what's the goose and what's the golden egg at some level you know we uh as an organization set out to develop back in 2000 institutional framework within which to make breakthrough innovations and create companies to to realize them to make them real and and if you go back to the late 90s let alone the fact that everybody was infatuated with the internet and and nobody really did much of this kind in biotechnology although the genomic sequence was being at least bit by bit revealed at the time the notion that you could systematize breakthroughs which is an offensive thought i'm sure to the audience even today let alone further systematize and professionalize the act of conceiving and creating a company around set breakthroughs not around yet another version of the same company that you're making with a slight variant the notion that you could systematize that first-in-kind companies first-in-kind technologies was was kind of obnoxious enough that it was interesting to me it was interesting to other people that i could assemble because it really was a scary scary journey and and that's what we wanted to undertake so 20 years later um we in fact have at least the beginnings of of an approach to making breakthrough innovations we call this emergent discovery uh and we've described it a little bit happy to get into it uh here but basically that methodology which involves making leaps and then searching around where you've left too as opposed to searching around where you start that plus a methodology by which we think about a stage-gated way to produce a company out of nothing those are things that have provided the the let's say frameworks that allow us to create and launch six to eight totally new platform companies each year uh operate currently some 50 plus companies that are all founded based on ip we've developed and so that's the that's what flagship pioneering is it's a if you will a company of companies but a particular kind of companies and that is ones that develop platforms that are based on brand new science for technology and have companies built around them and i would absolutely encourage people who haven't made themselves familiar to make themselves familiar with the processes that you've described i've been made aware of a range of hard business review articles there's a ton of information on your website and i will take you up on your invitation of maybe digging deeper into that process around i think you describe it as emergent innovation or emergent discovery perhaps in a moment but i want to come back to that notion of mindset um that you've spoken about in the past as well around what's the mindset that you have to have in order to almost push away the standard approach to things and you've spoken uh previously around this notion of paranoid optimism i wonder if that's one of the mindsets that you personally have or the ones that you look for when you're building out your team but what are those mindsets that really are required for flagship success and its innovation process well there's several things that i can just very briefly mention starting with the paranoid optimism part yes for some 25 of the years that i've worked um you know first time i had to talk about these things um was when i had done it a couple of times and people wanted to that forced me to describe things and i always hated listening to people's stories about how they started a company because it almost seemed kind of like you know random in the sense that they were telling lessons but they had an n of one i mean they had one example and i thought what can i say that's a little bit beyond that it's a bit more repeatable and and this notion of paradigm optimism um is is one that really rose to the top for me i was aware of andy grove's book about the only the paranoid survivor i remember when it came out i actually met with him uh back in that time frame and saw him give a talk and and and i thought you know paranoia enough isn't enough to do something that's never been done before because it's depressing uh and and because you know that's usually the psychological state you'll get into um but but if it's if it's juxtaposed with a more daring exploratory mindset which is optimism optimism basically suppresses your concerns paranoia heightens them and if you can toggle between the two simultaneously not you know kind of being either optimistic or or paranoid but simultaneously being a lot of each that struggle ends up causing you to be poised but still action oriented right so if you're not optimistic enough you just won't be willing to make the leaps that it often takes to do something truly novel if so you're not paranoid enough you'll make reckless leaps and not constantly be re-evaluating and adjusting based on real world evidence of what's going on and so that that juxtaposition is an important one but there are other mindsets that i've come to view over the years more recently i've talked quite a bit about this notion of an immigrant mindset you know if you think about it immigrants you don't have to be an immigrant to have an immigrant mindset but having an immigrant mindset is a very important part in my view of being innovative in the sense that if you think about an immigrant who changes countries either by force or by uh aspiration wanting something better for themselves what what you find is somebody who is unaccustomed to the to the uh habits of where they end up barely knows the language often doesn't take anything for granted doesn't think anything is old to them and is constantly worried about the police or the rules or the violating the law they're just in this highly paranoid state and they recognize that the first order of business is to get out of a survival mode just to survive and then try to kind of proceed and so that survivalistic mindset is really really important but one of the things it does is it actually helps you take relatively more risk not less because if you think things are owed to you and if you think you're supposed to kind of say authoritative you know kind of believable things you're not going to be in an immigrant mindset you're going to be a a native and and natives you know over time lose their edge as to willingness to do the dirty work willingness to to be willing to fail so you know if you're an immigrant you start out failing and your only better alternative is once in a while succeed if you're not and if you're not in that mindset you actually expect to succeed which is which is a disappointing state to be in in my experience when you're dealing with novelty and startups etc so i think that's an important mindset as well then the third mindset uh that i'll just share uh is this notion of how to deal with dogma how to deal with expertise you know when you're doing science when you're doing technology expertise matters a lot and of course there's this notion that experts know a lot and that's true but what i've observed over the years is that experts know relatively little about the future what they know a lot about is the present and the imminent present the imminent future that's the adjacent future and in fact they over index with their authoritative knowledge about that such that if you actually ask an expert about a breakthrough actually their general orientation will be to be highly skeptical especially if it's contrary to what the conventional wisdom is that's what dogma is all about so i i've come to view uh dogma as kind of a form of intellectual gravity that's very very hard to escape and so we're brought up in academia in in in science in general respecting the the the the college of study that we're in the area of of study that we're in and people generally don't say unreasonable things because they'll be chased out of that field and yet i would argue and we'll come back to this that most real departures most real novel thoughts start out being unreasonable by definition otherwise they're not a departure and and yet the rest of the world whether it's nih grant proposals or or corporate funding or venture capital funding what is all that about going and asking experts what they think of your otherwise remote thought and what are they going to say bad idea won't work if it could work it would have been done already and i really don't think you should spend any money on that so so that gravity brings you right back to earth and it forces you to find an ever so differentiated tiny incremental thing that people will bless on the one hand and yet might give you the opportunity to create value so how you overcome that is is kind of to make peace with the fact that if you truly want to do things that have not been done before you will be attacked by definition by dogma amazing i mean the way that you and you've clearly had to hone in since i'm sure a number of conversations over the last few years around this articulation of mindsets and it rings true to a lot of the work that i'm personally doing and i'm sure a lot of our audience is doing too and when you think about breakthroughs i i know you use the word but i know you don't take it lightly it sounds like flagship is designed to not just think about one breakthrough but multiple breakthroughs across all multiple areas of healthcare and i wonder if you wouldn't mind just sharing some of that because the approach you have something that i've read about certainly um to systematize that seems extraordinary actually the way that you as i've heard it look at impossible problems and challenge people to even extend that their thinking leveraging the tensions that you've described would it be possible just to talk a little bit about those processes as you think through that process to to from uh the idea state to the the prototype company to the scaled company sure i'll i'll try to do that a little bit and it's always frustrating to the audience because it's not enough but i would encourage you to kind of just look up and read more about it because it's something that often resonates with people but not in its kind of formal formal way of thinking about the steps involved in doing it so you first have to start with a a change in belief system that that i would argue people should go and do their own research about which is that i would argue that most if not all breakthrough innovations start out life in a relatively ugly you know kind of form and that they're not they know they're not born beautiful they're not born functioning and so if you're trying to find a starting point that is already in its full form a major breakthrough that's an illusion so i have come to believe breakthrough innovations as being largely an emergent act i'll point out not to offend anyone but including the work i've done often what happens is the people who are there when that emergent process ends up with a beautiful outcome that the people who are there at that time take full credit for how it happened and in fact describe it in ways that are highly uh um revisionist not not in a malicious way but rather in a rationalizing way i always tell people you know is what you're describing the rationale for what you did or the rationalization of what you did rationalization has the benefit of hindsight and i find that often the way people describe innovations is by knowing what the end of the movie looks like and boy is that a different uh experience than the way it actually happens just my observations over 35 years and i've met hundreds of other people who've who've engaged in in doing things that you know defy gravity somewhat so first thing you have to do is to kind of say you know what so this is not a goal-based activity this is not something that is not designed you don't design a breakthrough it emerges so if you believe that the way we've come to think about it is that there's two things you need to think about of the processes we know of that are say emergent in nature that is a non-linear combination of of the parts iteratively ultimately give advantage darwinian evolution is a damn good one uh it's produced us it's produced most of the life forms uh that all of the life forms we know of all most of nature and so and so if you look at what is that that's basically just variation selection and iteration if you take those operators and apply them to dna or any information molecule you get optimization you get improvement and optimization of fit between the environment that the entity is living in and the entity itself if you by the way do this in software you'll get all sorts of you know machine learning type generative algorithms that create you know beauty create novelty that is an unexpected outcome of the steps you were taking because it's truly emerging so we knew that 20 years ago in fact i worked on a company is what was called affinover for many many years where we used evolutionary algorithms online to design products of all kinds using consumer feedback this was back in 2002 3 4 in those time frames early days of internet so why am i saying that we started using that mindset for the kind of work we do so i said there's two things one is that but it turns out that if you apply that to ideas that are proximal which you could you find yourself applying a powerful emergent technique in what is otherwise a commodity space and you might say what do you mean by commodity i would argue among the things i wish i knew 25 years ago is that adjacency oriented innovation which i'll argue is what most most of us do adjacency oriented innovation is by definition a rapidly commoditizing space because incumbents preoccupy it literally preoccupy it it'll preoccupies them venture capitalists occupy whatever is left over because they want the incumbents to buy out the companies academia is because of the way science is funded naturally occupying adjacencies and gradually kind of advancing them so the good news is you can apply a new technique the bad news is it's a really crowded space so the part that was kind of rather a departure even for us years into our existence was the notion that what we needed to do is to escape the adjacency and so how do you do that and that kind of you know for lack of better words kind of maybe people think about it as a leap of faith you need to be able to leap you need to be able to create hypotheses of things that could exist even though there's no proof that they can actually happen and the value that would derive from them if only they existed and in the first instance gennade and this might be a little odd to your audience it's purely an act of imagination you know we can call it brainstorming we can give it lots of ideation there's a lot of euphemisms for imagination it's just the imagination so the beauty is if you have some experience and you're uninhibited you should be able to imagine things that don't yet exist you people should have been able to imagine and i'm sure they did the cloud of cloud computing decades before what what stopped people from imagining in fact there's a whole field of literature called science fiction that's all that is is this reasonable descriptions of possible futures well if you do that it turns out in in the problem-solving space or in a way you seek impact you could imagine products you could imagine underlying technologies and by the way you have to keep yourself that's the paranoia part realizing that it's not real just because you could imagine it but that gives you the ability to leap and leap beyond where people are working now now when you leap you'll end up with some starting points that's the initial hypothesis we do this maybe some 80 to 100 times a year we conduct these formal explorations where we generate hypothetical value propositions in different aspects of health and sustainability that have no basis in current reality but they're the product of this leaping when you combine the two ideas of leaping and then using darwinian evolution to search locally and iterate iterate iterate so you might say where's the selection function come from so the selection function comes from lots of discussions with experts by the way non-experts even better people who have done startups people are in large companies people in academia so what we do is we take ill-formed imaginary descriptions of future realities we expose them to it's the best we can do the current set of thoughts about what what why it's a bad idea by the way experts are by far the best way to rapidly figure out what's bad about your idea they're like professionals at that and that saves you a lot of time you don't have to read all the papers you don't have to repeat every experiment they know why it's a bad idea so we use that and ultimately what happens is that without doing a single experiment you can a priori rule out a whole bunch of versions of what you're trying to claim could be valuable even because there's some fatal flaw in them leaving behind a few that aren't known not to work but if they work they could be valuable that is the step that leads us to then form a small what we call protocompany and go in the lab and do reduction to practice experiments which is our last art by the way you know patents are all about reduction to practice but nobody actually reduces to practice things because they don't use imagination they use more like discovery or design as mental tools but here what we do is we imagine we test as much as we can dry experiments discussions and then we'll go in and say can you actually even do this and if i map this on to moderna some 11 years ago summer of 2010 we wondered what if you could actually inject an information molecule into the body and have the body make just about any protein we wanted that was the formative question we knew that there were different ways you of molecules that could kind of give this information but none of them had been used as drugs before and so long before we did experiments we had to ask ourselves okay well if so what would you do where would you inject it how would how to get into cells how much protein would it make is it the right protein what diseases would you apply to you can ask all those questions without knowing whether you could do it to begin with and that's some of the thinking that we've used to allow ourselves to leap then to search for descendants of the original bad idea to come up with better ideas then to reduce it to practice if it's possible if it's not then it's really easy for a small amount of money we can kind of park the experiment and see if becomes relevant in the future and once in a while you get breakthroughs amazing and i love what you said about almost the practice of imagination there's one of the one of the books behind me that i carried in my uh pocket throughout all of my uh residency as it were or my house jobs in the uk and i never looked at the first page all i ever used it for was to look at all the clinical pathways the first science sentence of that book says medicine is a practice and an exercise of the imagination um because actually the essence of medicine is in my mind something that gives people hope it may not ever deliver a cure but actually everything that we're doing is a little bit sci-fi it has never been done before and everything that happened during the pandemic was rapid translation of science into outcomes for society and everything that we'll do from here on in will continue to exercise that imagination in my mind and if i was to give you a crystal ball newborn and think about the next five to ten years what do you think are the the what are your thoughts just on the future of health over the next five to ten years what are the the big challenges that you're in particular interested that society finds solutions for yes is the answer to that question there's no there's no shortage of both solutions and problems i should say a couple of quick things that come to mind that i haven't mentioned before one is that in this type of of what this by the way everything i've described is why we use the word pioneering it's you know unfortunately innovation is a commodity term in and of itself now so we kind of had to say okay well what is what is first of its kind innovation what is what is something if when you try to um you know bring forward a brand new value pool uh that people have not tapped into before and we call that pioneering in the in as a nod to the ecological definition of pioneering which which there is one in the dictionary which is kind of the organisms that first go into a barren place and make it livable colonizable so it's the first to colonize and we kind of thought okay if there are these barren places of value if you can find it first and access it first then you also get to set up the rules for how it's how it develops and certainly in mrna technology we've done that in many of the microbiome fields we've also done that having started in first companies and there's many many other places where we've played this initial kind of colonizing role so i i say that because as you go as you look forward uh the second concept that we also haven't talked about which is really really important in everything we do is this notion of of platforms we call these bio platforms and and while this is a well-known concept in the software or app kind of based industries or service uh in in the modern kind of shared economy where there's all sorts of platforms that enable multiple things to be to to be enabled at the same time in the in in the biotechnology field platforms are thought of as technologies you know kind of sequencing or pcr but that's not what we mean by platforms well mima platform is a common set of capabilities that have not existed before but that could directly produce many many many products and so mrna was a platform we did not develop it as a product we developed it as a platform the reason i say that is as we sit here today there are literally a dozen totally unprecedented platforms that we're developing and others are as well that i think will create whole new branches of medicine that either will allow us to go where drugs have not been able to go before or will will create a whole new modality of medicine even if somebody has done something before that gives huge advantages either in ineffectiveness or or in in in in time to develop etc so there's one set of futures that will be enabled by new and new platforms and new and new capabilities and and that is working very very effectively uh as we as we speak the other area uh which which you're kind of inviting me to speculate on is you know what can be done that's truly disruptive in healthcare and for me the the the at least the big uh promised or unpromised land i guess i call it that right now is uh what we've kind of been calling preemptive medicine or health security and and and the whole idea there is that um you know today we have medicine focused on disease forward we have generally wellness and and fitness things that are trying to preserve health and in between there's there's really nothing i mean there's supplements and the like that that are considered unfortunately not considered in the way the same way as medicine is and therefore most people don't really know whether to rely on it or not but but what's changed is in the last 20 years a lot of the molecular techniques that have been developed in service of disease-based health care but has never been applied upstream where there isn't a quote known disease so we've been trying to focus on this is even before the pandemic but of course the pandemic has made it a bigger issue at least in the infectious disease side but i think this applies everywhere that i think that that in the future we're going to have to use the advanced science and technology we have to identify pre-disease states across every single disease area and by molecularly characterizing them also intervene at a pre-disease state and and these will not be treatments because you don't have a disease therefore they will not be necessarily regulated as drugs because that's not what they are but they do for a subset of the population that is in a risk profile that's already advancing to a disease we need to intervene then and we see this in certainly neurological disorders where clearly the more advanced versions are extremely hard to deal with by the way cancer where most everything's been done for late stage cancer because much easier due to trials but yet the problem is much much bigger upstream so creating a field where medicine becomes preemptive there's this whole notion of prevention and vaccines but i think preemption kind of says some of this is is is going to happen but if we could delay it alter its form and and deal with it before you get into the disease treatment mindset you know i sometimes say you know if you turn on the tv you can't go one minute before seeing an ad that is embellishing a patient journey if it was up to me i'd rather embellish the pre-patient journey i'd rather stay on the pre-patient ride for as long as i could and then become a patient but unfortunately there's no such thing as a pre-patient you're well and then you're given a death sentence i mean i'm exaggerating and obviously you you know this field extremely well so i am at least imagining dreaming of hoping for and working on together with the whole organization at flagship a set of applications of our technologies and platforms upstream and hoping that by use cases by examples and impact we can get regulators and certain countries that are thinking in advance to begin to adopt some of these and realize finally that the best way to to reduce cost of healthcare provisioning is to move upstream we know this with vaccines they are by far the most cost-effective health interventions we have and yet up until this pandemic we had very few of them they were a commodity business nobody wanted to be in it and we'll see what happens after this after the pandemic but even beyond vaccines vaccines to me are the tip of an iceberg of saying it's too late when you're sick we gotta deal with these things upstream fantastic and as as you talk a lot of that through the notion of the question around data data access this notion of not just the medical data but the social determinants of health data your genomics your individual behaviors everything that could be defined as preemption kind of comes in and as i think that through i wonder what as maybe as in terms of closing remarks just your thoughts or indeed recommendations to the tech sector in general and how you think the tech sector should be thinking about their role in supporting life science organizations be it flagship be it large biofarmer and others do you have any thoughts around what the tech sectors should be doing or indeed be doing differently in the future well i mean clearly the health sector is is super data rich and and and is is not as advanced in its digital disruption if you will as many other sectors a lot of reasons for that that are practical some of them are legal practical meaning different systems incompatibilities etc but i'd say if you get into the kind of discovery research life science side of this it's even more the case that the techniques we're developing to make measurements are producing results that especially when integrated are completely incomprehensible to humans and i think we're well in an age where without artificial intelligence machine learning approaches we will literally be ignoring euphemism for that is abstracting the data uh in order to make sense of it so the human sensibility is going to be offended by the amount of data we can generate and and we'll try to kind of force it into compliance by only focusing on the things that interest us as opposed to leaving it intact and using approaches that can kind of give us some level of patterns to focus on versus just what we what we decided was going on i say that because i think we're just in the beginning of that and and certainly that's a pervasive aspect of what we're developing new companies around this several of our companies in the last two three years are entirely based on things that but for machine learning approaches and some data we could not ever think of doing and i think that's going to teach us things about disease mechanisms and pathways we didn't know about we're discovering entirely new states of cells in our bodies that nobody knew existed we're discovering completely new proteins we're discovering things you know we're discovering viruses in our bodies that nobody knew existed and you can't catch them without having a massive amount of data that you then start looking for certain things in by identifying certain patterns that you've seen elsewhere indicate the presence of this let alone generating proteins we have we have a whole whole platform we've developed over many years to de novo from scratch generate proteins of arbitrary function arbitrary binding ability so all of this is in the future and i think the tech companies who are on the one hand very accustomed to handling and and and processing large amounts of data and extracting meaning from them are going to have to figure out how to make that useful that capability useful and of course it's attractive to do it in healthcare simply because it just looks like consumer data at some level it's a different kind of data as you get more into the science i think people kind of have gotten a little bit nervous about it because they feel like they're at a disadvantage not knowing the biology but i think we're at a time where the biology will give way to the data science because it's again beyond human comprehension at least that's what i believe and it's going to have to be bilingualism not one dominant language then kind of serving the other and and that's that's what the next decade will be about newborn my sincere thanks to you i wish we had more time and i'm i'm sure i could go on for hours having this masterclass with you and learning from you and and your and your wisdom and my thanks to you on behalf of the research summit community here i look forward to hopefully connecting with you in person in due course and every best wish to everything that you're trying to do within flagship uh and all the humanitarian causes that i knew how you somehow find the time to do in your spare time so thank you again look i i appreciate the discussion i will caveat everything you just said by the warning that i told you that most of what we work on are things we know nothing about you should include the last 30 minutes as things have talked about that i really don't know much about but they really should be rather invitations for the audience to think differently so master class you know if if anything it's master class in in in improvisation not in knowledge i don't think this is a yet a masterful kind of field but i hear you and i appreciate the the chance to share some thoughts and hopefully people will have their own thoughts and and and start iterating around them thank you again