Fireside chat #2: MadeWithML.com -- Teaching Practical Machine Learning

Key Takeaways

hi everyone welcome to our second fireside chat on teaching practical machine learning uh with with goku mohandas and myself uh hugo bound anderson we'll get started in a couple of minutes uh but i just thought as you're joining now um perhaps you could introduce yourself in in the chat and let us know what your interests in machine learning are and what you're excited to hear about today it looks like we have someone called the wolf of wall street here that's that's incredibly exciting hi wolf we have op1 kenobi here as well um and we have utterly miguel from angola oh fascinating uh wolf of wall street um has said i'm totally new to machine learning but studying neuroscience um which is a topic i mean health in general in the sciences are topics very close to to my heart and even closer to goku's so that's really exciting all right so we'll get started in in a minute um uh just as a few more people roll in but if you if you'd like to introduce yourself in in the chat and let us know about your interest in uh machine learning and particularly practical machine learning uh it'll be great to hear uh why you're here today foreign all right well it is time to get started so i'm going to share my my video um and stop sharing my screen um and goku is going to join us hey hey goku how are you man good good good to see you likewise um it is an absolute pleasure to be talking with you today about practical machine learning um and all the all the different types of things and evolution of things that um we we feel we may need to know um and trying to define help define contribute to the space which has a lot happening in it at the moment oh absolutely um i'm happy to be here i feel like we had a version of this when we first started talking a couple months ago and i i'm very happy that we're putting together putting this together more organized fashion now because i think we got to talk for hours um so under about an hour today though well it's interesting we have been talking for several months and i feel like the space has changed even in that time so i mean this is one of the reasons i wanted to speak with you today publicly is we can talk about tools and techniques and all of that but i think maybe what we need more are like ways of thinking and heuristics and approaches and mental models for what's happening and that's something i i really appreciate in in in your work but before before we get there i would just like to introduce you um and you you'll say a bit more uh yourself but um everyone um goku mohandas um uh is is founder of made with ml which is in in my humble opinion what one of the most exciting and interesting resources for lots of different people to learn about practical machine learning um in particular one thing i'm really excited about it or let's say two things i'm going to limit it to two is that you make very clear that ml is not a separate industry i'm going to read your words instead it's a powerful way of thinking about data that's not reserved for any one type type of person so you've you've got personas such as software engineers college grads data scientists product managers um in made with ml the the other thing that i do want to get to that i'm very excited about is when you when you start looking at mlx you don't begin with hey how do we you know build bags right or how do we think about constructing pipelines and that type of stuff you think about purpose you think about products system designs the actual projects um so it's very um upstream of of a lot of the symptomatic concerns that that people have and that's something i'd really love to dive into so everyone that's one of one of the reasons i'm very excited to have goku here um but goku has also worked on machine learning and uh product at a large company being bing apple a startup in the oncology space citizen um and has run his own uh startup in in the rideshare space being being a hotspot um i will also say we are having a q a um feel free to ask any questions in the chat everyone um and we'll get to them uh if if possible but we are having a kind of a week-long async ama on our community slack afterwards so if you ask questions there for goku and myself we'll jump in um every now and then uh and answer them so that's the slack link uh in the youtube chat and there's a channel there called um ama guests um so if you want to ask any questions there after the fact that would that would be super cool um i also just thought to let you know um i'm so i'm hugo bound anderson i i run developer relations at outer bounds um and at outer bounds where we're young and and somewhat stealthy um but we're just really excited to be working on infrastructure and productivity tools for data scientists that essentially uh allow them to focus on building models and doing science while having easy access to all the infrastructural layers i mean my personal having worked in basic science research and data science for over a decade now scientists i think unless they're really excited about it um shouldn't need to know all the details of compute and orchestration um and infrastructure but should be able to easily access them so it's building tools and wisdom layers and productivity tools around these types of things that we're really um excited about uh we're doing it through an open source framework called metaflow but we're also really excited to be working on on products um and i know you've been excited about metaphor as well goku so that's one of the ways oh 100 i was just going to quickly say i i don't do a lot of podcasts and interviews but i was 100 into this because i just love talking about this stuff with people that share the same mindset and there's not enough people talking about this we always hear the landscape is saturated with too many tools and platforms but it's actually the opposite i don't think there's enough people thinking about this in the right way and building the tools to kind of make it easier to practice those things so that's why i'm super excited about metaflow and outer bounds and i've met with quite a few folks in the team over the last couple months and uh yeah super happy to be here awesome so um if you're watching and you're enjoying this um definitely share it with friends and hit subscribe if you think it will interest them as well um but without further ado um i'd love to maybe we could set the scene by hearing about you you goku i'd love to know a bit about your background perhaps with a focus on what led you to build um made with machine learning sure um so you kind of laid out the the big milestones there but definitely i think in during each of those times i never knew what the next thing was going to be but my background uh it was is in biology and chemistry and i saw some folks from neuroscience watching right now so i think it's amazing that this field and sorry this practice uh and the field of technology itself isn't just being kind of just reserved for the things that make short-term money right advertising finance et cetera and there's there's people in just amazing industries life sciences energy uh geography just using this uh to kind of accelerate everyone's respective industries so i love that um but that's my background in in the biology side and if we have time at the end i kind of i can talk about uh my main thing that i'm working on now which is on the genomic side of things but yeah and i actually didn't learn how to code until i say the last couple months of undergrad uh and i always share that with people who say it's too late because you know now i work with people who've coded since they were four or five right uh sometimes their first language if you will um so it's never too late for people who are thinking about uh you know jumping into tech and specifically trying to leverage ml uh in fact there's the pros and cons there's a lot of resources now but um you know i think uh kind of once you weed through that noise it is very possible to learn and learn quickly and come into this space but like you said i had a starting backwards i had a startup during grad school in the ride share space it was my first foray of applying machine learning in production um this was seven years ago so it looks very different uh and uh maybe slightly embarrassed about the systems i built back then but i love being able to build something for a specific audience and then actually being able to maintain that over time uh and some of the principles that i learned there stay true today and uh you know i definitely tried to share that i made with my mouse others can think about it the tools have changed that industry actually has changed completely ride sharing uh you know uh but the principles that we used to build at that time uh are actually lasting so that's something we'll touch on later um uh after that um unfortunately we were on the wrong side of the market we were we would predict where surge would occur for the taxi market um and at that time uber and lyft actually had something very similar already brewing up uh but the very very close name actually as well but then i went on to work at apple um mostly nlp for almost three years and at the tail end started to come back to my domain which is in the health space uh and i'd left to join the director of apple health records after his younger sister passed away from cancer complication he started a company in the oncology informatics space and you know ml it was no one's vision that ml is going to solve everything but it could potentially play a very crucial part in sort of reducing the amount of data that um an expert would have to process so that was kind of the gist of what we built there um i had an amazing time there and uh while you know the space is still growing nothing's truly solved i think we were able to push a very small aspect of that and there's so many companies in that space now which is amazing to see but after uh we were acquired i had started uh hearing whispers of this thing called covid and i didn't exactly know what i wanted to do next i built ml systems at a large company built it from scratch at a startup for a very specific industry and the start of my own as well and i just wanted to share kind of the commonalities across these three different experiences and that's kind of where made with ml is right now it's it's uh sort of this basics first principles approach to thinking about a lot of these topics and i try to make these things hands-on as possible because it's not enough just to read about this but you need to be able to practice it and exercise it and actually continue to do that over time as well so you keep sharpening these things but that's where we are today um or rather about six months ago uh kind of continued to add content to made with ml and we could talk about um kind of my focus there but yeah it's been kind of a crazy journey um but uh definitely uh looking back don't regret any of it and it's definitely all influencing what i'm working on now uh and hopefully continuing to be working on what a wild journey and and very exciting and inspiring journey as well i i do want to kind of dive into some of the mechanics i've made with ml but there are a couple of things first i just totally resonated with with me when you said um you you learned to code and later in life i'm in a very similar position i didn't really start coding until i was doing my my postdoc i'd moved from pure math to to applied math working in cell biology and biophysics um and my my job was ostensibly to do mathematical modeling for the biological sciences for cellular dynamics actually cytoskeletal dynamics like mitotic spindle stuff and that type of stuff but i was working in a biology lab with a lot of biologists who were generating gigabytes if not more um of data a day and it became i'd done some statistics and you know i'd done a lot of math right um but it became clear that i that i needed to figure out how to wrangle all this stuff um this was in 2011 and i i discovered something that was then called the ipython notebook um and of course we know where where that that journey and journey taught us well that's actually how i ended up in education as well i love doing research but i started splitting my time between research um and teaching what i call practical data science for research scientists workshops um at the universities and institutes i worked out and it became clear that i could have more impact that way actually i enjoy it both the same but the amount you you mentioned noise before and i think um one way i like to think about this is um we're in an information abundant landscape now um and for developers scientists people wanting to do research people wanting to put things in production um we need to increa increase signal to noise for them right yeah absolutely um i think things like this podcaster are going to be doing exactly that um just really quickly i your experiences resonates i think with me and with so many other people but the need to learn something is is different from being forced to learn it so i said i started learning to you know learn to code at the end of college but i actually took a class in high school and it was kind of in a force scenario uh wasn't really the mindset just learned it to learn it right or get through the course uh sort of a requirement and it really didn't connect and then you know years later when i needed to use it for something and there was a reason behind it uh i was i enjoyed learning it um you know to the point where i continued doing it even after fulfilling the immediate need but i want to touch on that as we talk about topics today because this whole uh you know topic of ml and ml ops these are just approaches to solving problems and it's one way to solve problems so you you learn about things in the beginning to get a good understanding but then you get into the details or explore other techniques when you need them and there's always an explore exploit that you should you should kind of uh a mindset that you should follow but yeah it's just so much more fun when there's a purpose behind learning something uh i i think you know that's that's how i would operate but yeah it just it didn't feel like i was just going through a book or something yeah agreed completely and i think that the need when you actually have something that you you really want to do and in fact um you know that i've worked a lot in the pi data space and the pi data stack um is very close to my heart and i always um make sure people are aware that it was it wasn't built for the most part by computer scientists or software engineers built by neuroscientists astronomers geologists people who needed needed these tools they noticed that you know fernando perez had built ipython that there was some cool stuff happening with matplotlib and it's like oh suddenly we can have this swiss army knife essentially um but the the need i my um my dad's retired now but he was um uh literature academic and writer and professor and he um so i get quotations floating in my head i don't even know what they're from or but i remember when growing up he would say that the threat of execution focuses the mind wonderfully i've got maybe oscar wilde or something something like that but when you need to do something right um yeah that's when when you really zoom zoom in um i just wanna this this may be dangerous to go here so soon but i'm i'm i'm i'm i'm ready i'm i'm feeling feeling comfortable um we've mentioned machine learning in production several times what what even is that yeah uh so let's dive right in there i think right now a lot of people um learning this stuff in school or you know maybe you're already working and you're trying to get into the topic you see a lot of uh model focused tutorials research and things out there right where you have you have a task maybe you have some kind of a toy data set but it's this closed uh learning scenario where you're just applying a model to some data set and you're trying to see something like a performance or something and you close a notebook and you say hey i've learned about this model i learned how it works i've seen the results on this uh you know toy data set i've learned it when it comes to machine learning and production um first of all there are a lot of assets and i like to call them assets because it's not just about data but there's there's so many assets like the the data the model's weights the performances configurations for the different systems you're building separating things into workflows and then to add to all of this these assets are not fixed these assets are now dynamic your data changes your your models will change your performances will change your maturity in terms of you know what the system looks like will change as you kind of build out and you need to scale out where you need more granular functionality so you have assets and you have that and these assets are dynamic it's just so hard to learn this because there's so much influx um and that's that's kind of uh one of the main things that i'm trying to address with made with ml uh is how can we teach something so dynamic in in a linear fashion uh and linear is key word here because even the processes are very cyclical um you'll you'll get you'll notice on made with amal i teach testing like in the middle of the project based course it really doesn't belong only in the middle like i i actually do test all the code that i even did in the poc notebook it's just that i've when you're working in industry these are reused components that i'll reuse but you know i'm i'm linearly i'm trying to teach something in a linear fashion because that's the best i can do asynchronously but it's really a cyclical fashion and there's so many assets that are dynamic uh so yeah it's it's it that's kind of machine learning and production is almost the opposite of pure ml where it was once something that was model centric is almost model agnostic now uh it's everything else centric data centric pipeline eccentric workflow centric product centric we start with product right that's the first thing to start with um but yeah it's yeah i think it's a that's a good uh revelation model centric versus now model agnostic yeah which is yeah that's how i would separate it i love that and that actually dovetails really nicely with where i wanted to to go next which was i mean we talked about pi data for example where a lot of the education systems or the tidy verse right a lot of education systems around this teach some ways of thinking but they're also dedicated to teaching apis and get people up and running as quickly as possible with psychic learn or tensorflow or whatever it is right um but compared to learning these types of apis there are just so many moving parts to working with machine learning in in production so how how did you even think about getting started and how would you encourage other people to to think about it yeah um i'll start by saying that uh i think no one has really figured this out yet how to teach this stuff um and i mean even for me the last eight months of work that i've done with uh several other companies um i have like a big to-do list of updates which i'm planning to do this month actually um so just for people watching um right but and at the end i'll share a few other resources as well including our next speaker has a great resource on this as well but for me the answer has always been uh project based you can never look at these things in isolation when you first start out you know if you just quickly jump the gun and look at something very complicated like feature stores you're going to be very overwhelmed because you don't know what is this replacing like what what what are the workflows that this is making better or simpler and then what exactly are the benefits or disadvantages i'm getting from this you don't have the you won't have that um the decision-making uh mindset the ability to zoom out from a specific tool and say you know this is actually what it's doing if you just start jumping into tools so when you start a project and you know it's better to start something simple uh where you you know you're not changing too many things so you know the model bit maybe we'll keep it keep that super simple in fact i even try to make it a plug plug-and-play type of model if you want to if then model sure keep that if you want the latest and greatest short you can swap that out as whatever whatever you need and then we'll try to keep the data somewhat fixed as well um but keeping those two things fixed uh trying to develop a project and you know really starting from the product side of things and then uh going down in terms of uh the maturity levels so the way i broke down the course actually um and i think this will stay the same is this uh increase slow increase in maturity so first you know you mentioned uh develop doing data science in in the local computer right in the notebook so i do that first actually even for that you want to think about what is the you know bottom line impact uh then what what exactly do i want to do in this notebook if i'm doing a poc what is my objective what am i trying to test uh and then after you have this poc then you want to think about the next layers of complexity and even some of the companies that i work with many of them are at the poc level what they see a lot of times is that oh they see this you know level three level four infrastructure from google uh and they'll try to kind of not copy it because it's very complicated but be discouraged by because they think it's either this poc where i'm doing handoffs and it's not very clean or it's that the super you know complicated monolithic thing i need to run on this thing called k8s um and but that's not true right there's this there's this path this journey that you need to motivate yourself for why certain things are done a certain way and eventually if you need it you will maybe perhaps use that complicated structure but for i think 90 of the people that i end up talking to that or that i see online um that's not really where that's not the type of scale of maturity that you'll need um and uh i know the next talk is on on this so we don't need to go into too much detail on that but exactly and i will actually just share the the link to that and maybe we'll dive a bit deeper uh later but i'm going to share this in the chat everyone our next fireside chat is with um a fascinating human and data expert um jacopo tagliabue um and it's on reasonable scale machine learning and the premise is that you're not google and that's totally okay and let's figure out how we can get roi on all the projects you need to do using using machine learning and it may not involve kubernetes um but we're gonna we're gonna dive into that in in in in the next one so i thought to show that a fantastic resource by the way i i talked to jacob a couple weeks ago and i thanked him for creating it because uh i think it's a it's a myth that needs to be broken down that it's either this this completely unruly thing or this super uh heavy overhead and it's not true at all for most people so yeah i'll be tuning into that talk as well um but uh just expanding a little bit more on this topic uh for how to get started um so if you're not jumping into tools and it's project based um your goal should be to learn everything from the first principles so you know why why are we doing certain things how are things done uh currently what are my tooling options and then actually get hands on a little bit and i'll talk about the hands-on bit in just a second but this first part is super crucial because when you're doing a project let's say you're doing it at work or you're doing a project at home these stacks are going to look so different so many times you don't have control of the stack at work right there's some legacy system or there's uh i've actually i've advised a couple health companies now they have a stack from day one before even getting the first hire because of certain regulatory concerns or it's just what you know the cto is already familiar with they've signed you know the licenses for it so you don't have control of this stack so if you try to become a specialist with just a certain tool and you you really you know learn all the functions and calls that's that's really going to break apart the moment your context changes even in the slightest um so it's really important to learn first principles why why certain things are done certainly how to approach this so that you know later on you can adapt to anything it's all syntax uh once you know these things you can adapt in a few days or even less um and certainly when you have to make decisions you know comparing different options different platforms different tooling things you'll know the right questions to ask along the way so that's that's kind of the reason to focus on this from a from a project standpoint um uh yeah that would definitely be my my number one recommendation there and then in terms of specific tooling you know if you're already working somewhere that has certain stack you know that could be that that's the stack that you get familiar with that's the one that you end up using with this foundation um but if you don't have something then you really have to look at your context and i think of it like a tree you know if this foundation is the seed there think about all the different branches or roots rather right that grow out of this um depending on the context your stack will look a certain way depending on the context later or certain needs your stack will look a certain different way so it's really important to understand the whole thing first and then go down very specific stacks and become specialized in that i totally agree and and one challenge i i when people ask me how do i kind of get involved in machine learning in production this is advice i know a lot of people in in in the space give but it's to get the one of the toughest things is getting the first opportunity and finding that at work for example getting a project where you can start your teeth into it so there's a tension here but like it isn't like you can do a leaderboard style competition and learn you can learn about psychic learn and pie torch that way but how do you learn about in production so there's a tension between being able to even do this when you don't necessarily have the opportunities so is do you have any advice for people who let's say they don't they're not doing it on the job but how they can get started in in in their own life it's hard um and i think uh you know efforts like made with amal and yakovlus repo there's actually only a handful that i can name right that even that even attempt at doing this um and it's because it's difficult and it's because of those dynamic assets that i mentioned we can start in the very uh we can start it's not the beginning the beginning is the product but after you've motivated things and you're thinking about even the data source right let's say you need to fetch data not from a single file but maybe it's from a warehouse or a database learning how to interact with these systems is is a huge endeavor in and of its own so what do you do here do you set that up on your own okay sure if you want to set up a database where can i go maybe i'll do something on roku digital ocean a warehouse sure maybe a snowflake oh but snowflake only has a trial license that i can only have up for a little bit it's it's so complicated to set this these things up um so at least so far uh the way i've been thinking about it is you can learn those maybe those tool specific things when the context arises so let's say your company eventually uh migrates to delta lake or something right you can learn the specific syntaxes to interact with how to you know ingest the data how to uh you know attach tests on top of the ingested data et cetera but for now let's try to do as much as we can on the local computer so far right so it's important to learn about these different uh i guess infrastructures and how to you know push and pull to these things but then assume that you have a certain kind of entity coming from this and then let's continue to work from there so the way that's kind of the approach that i've taken so far version one the version two that i want to get to uh you know this year as i have some time is to expand on that so again the foundation has been set now let's actually use some of these tools uh and the heart this is the hard part which becomes hard to replicate at home um but i'm starting to get a lot of these companies involved who are providing these things to have this educational layer so it becomes easier to set up it doesn't become something that you know i'm just just drowning money on just to have an instant setup or something sort of a sandbox if you will but those those dynamic assets are very hard uh especially on the data and compute side of things um i think those will change over time uh and you know our efforts will hopefully uh make that pave the way for that um but yeah that part is difficult which is why i i highly recommend that if you are someone who's looking to learn this uh you do join a company that is going to motivate you to learn this stuff so i have different different advices if you're a beginner um and again i i'm going to regret saying this but i'll i'll put the caveat at the end if you're a beginner i highly recommend joining kind of a midsize slash large company because you're not going to be in charge of the entire end-to-end stack right they're not going to throw that on you you're going to be specialized you're going to kind of uh learn how a specific part of the system works and then as you build expertise you can't you you can kind of expand from there uh and you sort of you know become this t-shaped developer if you will so by doing that you can get insight into the entire stack so that later on in your career when you go to a smaller company or you found your own company you kind of know how everything works and you have this experience to be able to to build this out but i yeah i highly recommend learning this stuff because you have the context at work to actually need to do this um and that i mean that's one of the main reasons that uh i think most of the people that took the course and reached out to me are people that are working at existing companies fortune 500s or startups where there is a need to do this and it's i actually don't know too many people myself included who do this uh just to replicate uh something on local computer and learn from it so yeah i think the need is need is key here um and i i think it's uh there's this talk by josh wills from slack it was a quote he said at the very end of his talk you shouldn't be doing this for uh just like the pure fun of it there's so much work that goes into this first building and there's even more work in terms of maintaining and updating it uh which we'll hopefully we'll touch on today those topics um that this is not a trivial thing it's a huge endeavor that that needs a lot of uh effort so make sure that it's worth all this effort uh and then to actually learn this you should be doing this in a scenario where you're not uh kind of over engineering things just to replicate something yeah yeah i agree with all of that and also kind of aligning on expectations i think there are two the two polls which we see occasionally are some companies thinking oh we're going to adopt ml because you can do anything with it and then at the other end it's people like hey we can do things without ml and finding you know part of this having these conversations is to find out what that space in the middle looks like and that's actually where i'd like to like to move now so firstly i mean you know you hinted at this but congrats on how many people have become excited about and interested in uh made with ml you've got tens of thousands of people around the world who've found it incredibly uh useful um and i just want to that that's really inspiring to be honest goku um okay but on top of that yeah oh yeah sorry go on oh i was just gonna say uh it you know it's not uh kind of my full-time thing that i do now but it's even more inspiring the amount of people uh during covert it was so you know i definitely spent a lot of effort during that time but the amount of people that uh still kind of reach out and you know i just i'm just providing the basics they it's actually really on them uh to kind of continue that effort and it's a whole lot easier when you have the right context at work or something to continue to expand on it um but the amount of in people from different industries that reach out uh it it makes i'm humbled by it because i feel like i've gotten to very slightly touch an industry or impact it and you know as an individual i never will have time personally myself to impact each of these industries but that's kind of my main motivation and it's all free for that reason and always will be free because i want as many people to learn this stuff take it to your field and accelerate whatever it is that you're doing um because i i i always feel this education layer should not be uh behind behind closed doors absolutely and it is i mean the education layer is as important as every other layer as well particularly in in this space i i think yeah um a really quick note on that that i had dinner with uh seven uh because one of the co-founders at our bounty couple months ago and we were talking about this there's too many companies in the field now in the in the landscape that just come in and throw something super sophisticated out there when the reality is more than 90 percent of the application builders who are you know your potential customers are not at that stage yet uh so there's this huge bonus on i would say a hundred percent of the tooling and platform companies to be a part of the educational efforts to get people to learn and think about these things and get to that stage where their their tool or product makes sense uh i think a lot there's a big disconnect there and people are wondering oh is this space too small or enough people not doing this no it's just they're just not at that stage yet and you really have to you you have to be part of this movement to get them there um very much and it could be on a client by client basis or something open that that's not that's not the main point but um focusing on that is key and and look at the packages that have for example and frameworks that have garnered serious adoption i mean psychic learn is one of my favorite examples and look at the education they've built around of it a lot of the time they practice documentation driven development where they write the docs first um and i think these types of things pie torture has taken off in a serious way the the past several years right and a huge part of that is due to their investment in the wisdom layer um yeah yes i think 100 so made with ml we were just talking about how a lot of individuals have found it incredibly um impactful i think there's a tension between enabling individuals um data scientists and learners to have access to everything they need to to build stuff in production and then enabling organizations to as well these these are perhaps quite different concerns that maybe are correlated in in some ways um but i'd love your insight into i mean you've worked with a bunch of fortune 500 companies among others on machine learning and production um so before we get into education though i'm just wondering what the distribution of machine learning maturity in these organizations even even looks like okay uh uh i will say i've been mostly shocked um i've been wrong about a couple things too i thought you know uh being in this privileged area of silicon valley everyone follows very mature techniques and methods across the board even here or you know across different uh tech tech silos if you will um the maturity uh there's vast diversity in maturity um so that was something very shocking to me um let's see maybe i can start off with like highlights and then the reasons as to why yeah um i've done a poll uh so i i teach mostly to specifically the machine learning teams within these larger orgs um and the maturity first of all i want to point out is not specific to uh the uh the market cap uh or you know or the company's uh company's market cap right it's whether it's a big uh it's a team a machine learning team at a very big public you know billion dollar company or a machine learning team you know that's starting from scratch as a startup they all have issues um and are all you know kind of somewhat low in terms of maturity levels for how they're thinking about things and i thought about um kind of why why we have this um because you would think a company with bigger resources they can just get things moving very quickly and have these mature at-scale systems built out that you really shouldn't be struggling with this but it's this mindset around machine learning is the reason for this um so maybe i'll start with how things looked when i first started this two years ago and then maybe some good news as to how things are changing now um but you'd see things where uh one one good thing that i've i have seen the last two years is where machine learning is applied um i think there's a better understanding globally in terms of what it where is it justified and i think that has a large part to do with a lot of the resources that have come out in the last couple years um uh you know around where is the male justified um i know a lot of folks come from like fast ai community things like that where they've developed this understanding of where is this really needed and where is this effort actually justified um so that that's that's been uh a silver lining that's been some good news there but in terms of how things are executed it's a very um individualized mindset i would say uh and maybe we paint a picture so there's a company that has this uh idea they want to do a poc first so they'll put usually one but sometimes one to three data scientists on it and everyone's kind of working in their silos so more than 90 of the teams that i work with don't like do not employ experiment tracking or any kind of management around their experiments and if you dive into it it's not because they don't know what it is it's because they didn't see the value of it and it's because they're individuals who are developing these models and then what's happening to these assets okay the model is trained on this version of the data set an asset is created and that asset is often handed off to a devops or an sre team to deploy it and the deployment part just looks like you know wrapping a little api around it and perhaps they'll throttle it scale it as necessary but the moment this breaks apart is anytime you want to update something anytime you want to look at what performance looks like if it's diminishing you can't do this because of this physical handoff that happened there's no reproducibility around this if there's issues around it you're gonna probably restart what you did and hopefully you can figure out what happened um but because of this ad hoc process uh there's there's pretty much no updates and oftentimes trust basically evaporates whether it's internal users that are using your application or external uh trust evaporates and the people who built this the initial version are off to their next project that probably will follow a similar thing um and i think this stems from kind of the the processes of software 1.0 where you're you know they were building something deterministic and it's passed on to a site reliability team who just has to manage you know what making sure it's up and if there are errors it's it's very obvious like what it can handle what it can and what should be produced um but this kind of handoff i feel right now we're not ready for in the machine learning world and this is what i was talking about in terms of efforts more effort to maintain what's happening around monitoring around valuation offline you know online and definitely post-production as well but yeah that's kind of i think this mindset uh it's starting to change and i think these resources are are helping push that but that's i think a large majority we're still in that mindset and because you know this there's uh trust evaporating and there's all these not so great results coming out of it a lot of folks are not uh open to kind of investing time into this space but again this is changing and now a lot of people do know kind of the right way to do it uh but now they they see that juxtaposition of this is what i have and this is what i should be having they see the google example but no there's all this uh there's all this journey in between that you should be going through um so that's that's something that i've seen um besides these things uh yeah i think as a whole um in terms of education again coming back into that like how so if this is the state of things how are companies working to educate people on this the really good examples i've seen are companies that are starting from scratch where they kind of are able to learn together and more importantly they're they're documenting what's what the process so that when teams when team members come in later on they're able to follow through the thinking uh and they're available they're able to catch up to why the current state of the system looks the way it is and they're able to continue so documentation has been key here um but for a lot of companies uh that are out there that do have some systems available um and are a little bit rigid i don't see a lot of education happening and honestly there aren't too many resources out there either to be able to do this so i think that's something that's going to have to change in terms of in the corporate side of things educating the current teams to how you should think about these things yeah i think we're just getting started with that too absolutely and i think you identified several key points in in there but one in in particular i think is you know um our friend and colleague villa toulouse who's the ceo of out-of-bounds and um works on medflow and and worked on it at netflix and i wrote an essay last year about the differences between mlops and devops and why we even need this new term mlops and we really focused on the introduction of real world data into into into software systems it isn't really just machine learning in all honesty it's software powered by data is essentially what we're we're talking about when you introduce the entropy and complexity of the real world what what changes there right and we see the convergence of a lot of data centric programming literate programming with with devops and figuring out what that what that looks like um the other thing that you kind of hinted at several times in in there which i think is a nice way to think about kind of an unnecessary condition for machine learning and production you spoke about reproducibility documentation hidden below these things is a bus factor of more than one right so essentially when we want to put something in production we want to make sure so once again this is necessary but not necessarily sufficient that if i build something in production i can close my laptop and go to barbados for a week and it's chill in the organization right which a lot of the time we don't even have absolutely right uh i think uh i think there's andrei garbathy in the last like tesla talk they had a metric for uh like days you can go on vacation uh and leave like your ml system alone um you know right now might be like zero days or something not a tesla maybe in most areas but yeah the goal is we want to get to that stage where we could put something into production go to vacation uh because you have the appropriate workflows work workflows in place you know ingesting new tested data being able to retrain your model have it run through the evaluation pipeline passes all the checks that you've uh you've built on over time uh so that this kind of this the the iteration needs to happen faster and with fewer handoffs but with more trust around it because you you've attached the right tests and workflows around it uh but yeah eventually we want to get to the stage where you could go to vacation and have it um just kind of show you hey i've updated and here's all the metadata around it that's relevant for why i updated it um but we're not there yet and i think it's uh it's definitely contextual that look the process looks different for every kind of industry and even data modality um and you know i try to focus on even keeping that agnostic but then when you get into specifics you need to think about how to specifically do that in your context but yeah i think that that is the goal um but uh we're definitely not there yet and i feel like in many places the number of days you can take off while to keep machine learning in production is probably less than zero as well yeah yeah to be honest um awesome so i would love to get into maybe some some practical advice um for people out there we've we've been talking around that you know one way to frame what we need here is drawing a bridge and connections and between laptop data science and what happens quite unquote data science on your laptop and what happens in the industry i feel like it's more like we need a bridge we more like have an indiana jones leap of faith that we have to do at the moment right um but for people who've you know done a bunch of stuff locally um and maybe start thinking about putting things in production what what are the top three to five things that you'd encourage them to learn and think about when when going down this path okay uh be i'll focus on the the technical side of things um i think hillary covered a lot of the the product side of things um which people should check out for sure by the way absolutely um i think first starting with the data you should have familiarity on how to necessarily set up because hopefully you know you have a data engineering team or some admins that have originally set up the infrastructure but how to work with uh warehouses databases uh and these terms are growing now there's a lake house there's there's all kinds of infrastructure but wherever the data lives you need to learn how to how to work with that and this is huge because schools still don't teach this um i have a little brother in school right now that did a program very similar to mine and the curriculum is the same uh and you don't learn how to work with data yet on day two day two of your work your pro your most likely is given here's where the data lives can you can you go pull that and figure out what's happening or can you go inspect it and see if we can do it with it or whatever it is and people don't know how to do that and that's that's really important to know how to work with where the data lives um and again this could look different across maturities right maybe maybe it is a file maybe or you know if it's super early maybe it's a database warehouse maybe it's already a feature store that you know the the team that you're going into has set up you need to be familiar with how to do it but again even before all that uh which is you know exactly what i focus on here with ml like what are all these things what are all these options for infrastructure for data how can we interact with them uh and then specifically whichever one your your company ends up using you you need to figure out the specific syntax and how to deal with that stuff it's so important this is why i think your heuristic of making things project-based is so important because it it's already overwhelming to think about feature stores and metric stores and data lakes and data meshes or like data this is just the data right exactly yeah this is like the foundational layer we haven't got to compute orchestration versioning architecture um yeah so i think the project-based approach getting started in that that fashion it stops the the deep anxiety that's associated with an overwhelming space essentially absolutely i think yeah absolutely um yeah you just reminded me i you'll notice that i the two other a couple other things that i'm going to mention right now they're actually all uh software uh principles that we're about to say right now you know they're they're deeply rooted in software that are applied to this context but uh there's a reason they're they're principles that have been adhered to over the last several decades um so it's super super important to focus on these things and to learn as much of this as you can you know if you are a new grad before you're out in the market because a lot of places expect you to know this stuff schools and uh outdated books or if there are books they're not going to be teaching you this stuff um the next one i would say hugo is uh definitely the habit of testing um and i've seen this especially in grad school uh i forget who uh is a prominent researcher in our community referred to it as like spaghetti code uh grad school students a lot of people like to write spaghetti code they know what needs to be done they know what their data looks like they just got to get this they got to get this function going they got to get this pythor function going and at the end of the day uh that's great just to like quickly experiment things but even for publish actually published research especially um and certainly when you're putting things in production everything needs to be reliably tested so that you're building this large system that's beyond the scope of what your you know one brain can store and theoretically like process exactly all the inputs and outputs how can you trust this larger system if you can't trust the constituents that make it so it's really important to get into this habit of testing everything that you do and people that come from software or learn about testing from the software perspective are familiar with testing code which is great but there's also the concept of testing the data that's incoming and testing the data not just at the very beginning but perhaps at the intermediate levels as well and then also testing your model itself uh which you know obviously has a lot of ties with evaluation and a lot of the work that you do here could actually be used for the monitoring side of things too which is kind of you know the online version of things that you're testing for if you will and then some more but yeah the the the concept of testing is so crucial and people think it doesn't apply to them because that's so that's uh you know software 1.0 that's uh not really pi tests knows that's not really for me but um no testing data models uh are super important um and it looks different it looks different for for for this context so that's actually i i cover i look at like stats uh for which lessons people are most interested in things like that testing is is one of the biggest because i think it's severely undercover for how to do this i think just in the last year and a half we actually finally have organized packages that are really enable this before this people were trying to uh use software 1.0 options to try to do it here but now we have good good packages great adaptations deep checks et cetera dq um to to do this for us which i see a lot by the way people trying to adapt software 1.0 solutions to do something for this context and data modality while the concept uh makes the rel you know follows through once you start to apply to this probabilistic scenario you can't use it exactly as it is monitoring is another one like i remember at apple we you know we try to glue together prometheus grafana on top of a stack to try to monitor things it's great for you know floats but the moment we start working with texture images and then want to monitor things beyond just these floats uh those kinds of solutions fall apart uh and you know especially like high cardinality and things like that so i think it's it's amazing that we have these uh options now that are created with the same principles of 1.0 but for this context um yeah and i do want to zoom zoom into that because i mentioned earlier we have um you know a very long tradition of um data centric programming particularly in notebooks i mean going back to like mathematica notebooks and literate programming right and then all these soft software principles on the other side and what happens when they meet and something i'm hearing here that i really like is we're bringing in testing principles but we want those automated testing principles to really i i suppose reflect the scientists approach from the other like the types of things we all did in notebooks um maybe not in an automated sense when we were when we were trying to import data in just a like pandas read csv right and then the types of things we did there to make sure it's what we thought the types of things scientists did in in lab notebooks when they you know glue their pcrs in or whatever and start writing notes and making sure that everything is what what you think it is or what you expect right yeah oh absolutely um uh i i i wonder if we're going to talk about this today um the larger topic but i love this is why i love open source work um you know the for example all the companies or packages that i mentioned just for testing ml they're all open source all their code is tested in the beginning you know even myself i was like why would i use this i already have my few tests but over time my context changes uh you know my scale changes i want to be able to use this community built package that covers what i need now and then something and and add you know i i'll contribute to it vice versa and it's all tested and and and trusted um which is what you want to do for your own code as well but yeah i love um i love that that that comparison that you made it's software 1.0 with how scientists think and then joining that um and creating something that very reliable awesome so what next what what are a few more things uh i'll definitely um workflows is a thing um i think this is this might be one of the biggest uh uh i think this is why i originally um we originally started talking right with uh with the outer bounds team but when people are working in notebooks they're kind of in their head it's kind of this it's sure there's many functions and classes and cells that you're working in but it's in my head and a lot of people said it's a monolithic thing that they built in this notebook uh i've you know processed my data put into this model evaluated it and i see the performance and then maybe i'll move these assets and you know put a fast api or flask api around it but the in the in industry uh i think maybe with the exception of a few companies like netflix and a few others um most people don't think about uh like this monolithic thing right or definitely don't directly can't directly translate the notebook in into production right away so the idea of separating these things into workflows is super important um something that you did in one cell very quickly let's say like downloading the data or maybe updating the data whatever it is actually warrants itself to be its own workflow because there's in production that's so much more dynamic now if there's new data coming in can you trust it how do you use this where does it go next and this idea of separating things and thinking about things in a workflow is foreign to a lot of people uh because there hasn't been a need to do that but in industry and in production that's everything um and again going back to going going on vacation if you don't have things set up in workflows like this that can each you know scale on their own and each uh consume things and produce things and each be tested then you you can't have a holistic thing that that runs on all those workflows yeah the concept of work workflows is definitely missing and and i think this does also speak to you know some other software engineering best practices like refactoring your code right so when we're working metaflow for example where we write bags um and and and directed by graphs writing the steps in them in particular is trying to decouple you know you start writing them and suddenly you know you're trying to tame complexity because they get they get pretty long right so then then beginning to decouple your you know execution logic from your business logic or machine learning logic right and starting to refactor certain certain things there so that your dags are essentially more more readable as well right and making sure that that reflects your mental model of your workflow also absolutely absolutely right um and i i'm sure you guys i know you have a lot of resources on this stuff already but i love the way how metaflow allows you to separate things even down to dependencies right for a specific function you need a different version whatever it is the ability to control things at that granular level and think about things in workflows that way um is super important um and you don't see that out there right and that's actually i would argue maybe the one or number two cause of so many issues it's like oh you know i have this conflict or this doesn't work with that and if you can isolate things and think about it this way you you it's clean it's it's clean for your for your sanity um but it's it's just it's it's it's a really good way to organize things um yeah well very much appreciate that and particularly i mean my heart rate just went up when you mentioned dependency management i i think this is still a story that we're all actively working on i mean i've got you know real yeah different i mean this is you know i wake up in cold sweats wondering about you know what what version of tensorflow i have in a particular oh i believe you environment it's brutal man um i try to control as much as i can but you know a lot of the things i put on co-lab because it's great people get a gpu they run it on there there are there are some periods where i nothing has changed i have been away completely uh code has not changed i come back a couple months later dependencies have changed internally there are some conflicts now and things that ran over and over don't don't pass like the third cell now um yeah and it's so frustrating uh but yeah that's why in the lessons we quickly move away from these environments that you can't manage to scripts uh and and conflicts that you can you can fully manage and and control um but even then there are some issues but yeah or absolutely better you mentioned the term spaghetti code before we also have the term dependency health and yeah very very good very good reasons the other thing i mean occasionally this happened a couple of weeks ago something wasn't working in my environment and i just rebooted my computer didn't change anything else and then it worked and i was like i honored like seriously like what i mean i've been to the genius bar at the apple store maybe 10 times in my life and eight of those ten times they'll like just restart it and see if it works and yeah and i'm like that's that's the state of genius in in modern society um i i have two more things oh we're actually at the end of time are we but but that's okay if you if you're happy going over a bit everyone seems really excited and engaged so i'd love to go over 10 minutes or so if that's if that's sure yeah uh yeah i i like i said i think we could do hours on this stuff i'll very quickly touch on a couple more things that i had in mind um i think evaluation is a big one and this goes to uh people come from software but certainly people in ml now this is everything right uh being able to trust what you've built uh from the workflow point of view but certainly from the evaluation point of view like how are we doing and i think as a whole that i've seen in industry and i forgot to mention this earlier as a whole in the industry it's very it's still very much at a course level and the even the best companies i work with will have only like a fine grain level at you know if it's a categorical thing at the class level or something like that where i'm seeing the cream of the cream and where people are able to make actionable decisions based on even finer grain evaluation or the folks that are thinking about you know not just using the features that are explicit uh in terms of how to slice it right so if i have classes maybe i'm slicing by the classes themselves but by the implicit features um and this could be you know your top clients your your sources of bias that you're trying to do you know based on uh maybe age race whatever it is but creating slices that are contextual and unique to your business that you need to watch out for and then continuing to add to those repository of slices if you will over time as you are you know as your as your systems are are put into production but um yeah evaluation again this has this has these deep roots with testing and monitoring as well but absolutely evaluation is key um that lesson is probably one of the longest on made with ml so i implore people to check that one out um yeah i think i think those are my last one i would say was going to be around problem scoping itself um largely around labeling uh but i think maybe we'll save that for another time but yeah there's even in my last company we just we we didn't use labels we were given just because that's how things were done before we looked at the signal we looked at what needs to be done now and it requires a little bit of forward thinking you know is this a hierarchy should i have hierarchical labels can i do i have the right signals to actually maps these labels what if i want to add new classes later what if i want to remove classes later how do how do i create a system and how do i architect it now so i can adapt to these changes later yeah especially if external users are using these suddenly their class changes or their class disappears like what implications does that have um and this again i think ties a lot to the product and problem scoping uh aspect of things but yeah these are these are just a few few of the things that people need to be thinking about very much so and maybe it would be nice to just zoom into labeling very quickly and i do i love your opinions on this i i feel like labeling is at such a nascent stage and we still rely so much on on on hand labels um even about you know scale ai has pivoted in certain sensors recently but yeah um i mean they they built a very serious business around hand labeling there's a joke that they called it scale and they built it around hand labeling initially yeah i i think but there are a lot of other techniques that are available that i don't think we've found a way to adopt and a lot of open source packages and companies um supporting them as well but we are behind the curve in different ways of thinking about um helping machines to label stuff and having humans in the loop um in the most productive manner there i think yeah i think as a whole as an industry it's it's a harder problem than originally thought um the landscape itself three years ago uh i remember reading any article saying hey we're doing it with human labelers now in time we'll build models to replace all of these folks and that's not that's not what's happened at all right uh it's it's very difficult to do that and it requires an entirely different school of thought for to be able to do this at best you can have models in the loop to make this a little bit better um and easier but yeah i think there that's and again that's such a such a big space now too and i love the amount of effort that's going in there around active learning um there's a great package called clean lab now uh from some folks from mit for i have all this and i'm sure it's labeled how much of it is actually can i trust which ones are kind of messy or mislabeled being able to identify these things it's all about at the end of the day not eradicating time completely spent on this stuff but how can we focus the limited time that we have on doing certain things and focus in terms of what should i label um but also like in terms of you know how do i make this faster better and how do i make this a workflow eventually so that uh i can get this done because a lot of scenarios don't have natural labels where you know the thing that happened is the label for example time series or something um how do we make this into a workflow how do we trust what's coming out of there uh you know overlap across labelers etc so there's there there's a there's a lot happening here which is which is great to see but it's going to be a while before we just agree we'll have this wrapped up as a workflow ready to go yeah and i mean you mentioned active learning but we also got you know forms of semi supervised learning weak supervision transfer learning um synthetic data generation for computer vision i'm fascinated in fascinated by um there's there's a lot of action there the other thing i'm i'd part of me hesitates to go here but i think we're at the tail end where you know we can start riffing on kind of slightly tangential things and i know this is a very deep in interest to you um i'm going to give the example of what um hand labeling um the handling hand labeling industry um and the labor market um i think there are deep ethical concerns there that we need to interact with more now i don't necessarily want to talk about that with you but what i do want to kind of get your thoughts on because i know you you're you think deeply about this is the potential harms done by machine learning um and kind of trying to figure out what machine learning i hesitate to use the term ethical because i think there are normative constraints there which i'm not entirely comfortable with but let's say uh machine learning which which serves all all the people who who are stakeholders in it and perhaps does doesn't doesn't do harm how we can can think about that for practitioners and end users and consumers of machine learning products i know that that could take we could talk about that for 24 hours right but yeah um maybe i'll i'll i'll talk about a very specific aspect of that larger topic um i think a lot of it has to do with the intention of the team that's building these these systems right they need to be they need to be very intentional about thinking about what does it take to even from like the labeling side right like what is this what does this effort look like who are we using to do this what is what does their experience look like and then every step of the way when we put this into production who's using it is how is that designed is is are the models outputs directly shown to the consumer who is the consumer first of all is it directly shown to them is there another layer of filtering that's applied to them when something goes wrong how do we know like who's to blame or even more important who's to who needs to actually take ownership and fix this thing um there's so there's so much intentional thinking and decision making that goes into this it's a big question you go right uh to be able to think about this stuff and just to add one more thing so much of this is could be way down the line um yes you know i i'm uh deep into the genomic side of things right now uh using using machine learning for genomics and biologics and small molecule design uh a lot of the things that are being developed right now in terms of implications could be decades later uh right in terms of what's happening who's responsible for this and being able to attach an event that occurred to something that's being thought out flushed out right now um yeah i i think it takes a lot of forward thinking and this is why uh one one aspect of the solution to this is the need to have so many different personas in the room and not just you know developers who can build the system in the room when you're originally planning this out you need to have you know name be it policy makers the physicians the the subject matter experts um in the room when this is being thought out uh and i i don't see many teams doing that um because it you know it takes a lot of effort to to get first of all get these people together and think about this stuff but that's that is that is one of the best things we could we could do to for problems because these people have the the the sort of the foresight to see what could happen and how we should design things but absolutely yeah it's a it's a huge topic um and you know i think there's even collections and repositories of like ml fails and each one of these we could you know we could break down and see exactly what happened yeah and there's a lot of figure pointing that happens right with the zillow use case absolutely um and you actually mentioned fast ai earlier of people interested in these type type of things um the fast ai blog um has some beautiful and thoughtful pieces on this this type of stuff by rachel thomas and jeremy howard and rachel's work in particular is super inspiring i love that you mentioned in intention because i i think what you're trying to build because of course if you're building to building something and optimizing for engagement that's going to have downstream effects on all all types of stuff right um if you want people to engage as much as possible as opposed to i mean facebook right they've decided and i don't know the details of this they decided not only to focus on time on facebook but time well spent quote unquote whatever of course one i think one of the huge dangers is that the things that are really good for us in some ways seem to be most uh evasive to measurement as well so when we're yeah no that's true right it's hard to measure that stuff you'll also think about who's coming up with these things that are to be measured like the company that's building the thing to make the money off of it should not i think my my view should not be the one that's coming up with this noble metric that's going to measure how they're absolutely yeah and particularly if they're entering the commons there is an argument that you know these networks publics aren't they're not just private corporations who you know because about the systems we built can operate in any way they want it they have become the town square and i think we need different mental models and different um accountability yeah ooh this could be really interesting for another day but um i am starting to see uh kind of pillars at least in the health field where if you're a company that's promising you know um promising is a product with certain functionality in a space of certain performance uh there are pillars that are coming third party pillars that are coming to evaluate what this could look like um and you know the fda is one layer eventual layer but we want to be able to catch things at a more fine-grained level that maybe will take longer for the government to adapt and things like that but uh yeah i i'd like to see more of these pillars and it's not it's not about bureaucracy uh it's about something that perhaps industry-wide is adopted and everyone agrees that this is we need to all pass these kinds of checks and tests that happen periodically because these systems you know keep keep changing they're in flux but i like this kind of movement but it needs to be done right because we don't want more uh kind of red tape on everything because that staggers animation but we don't also want like you know free form uh things to just happen untested yeah and i think um initiatives such as uh uh model cards and this type of stuff really help us go down this path and you know i don't think this is quite the right model but you know we've got like nutritional information on on on cereal and stuff in supermarkets so just imagine whenever you're interacting with a model now i don't think this is actually this is fantasy in some ways but it's interesting to think think through whenever you interact with a model um having some information as to what's actually happening there because yeah something we don't talk about enough i i think are probably like machine learning dark patterns right in the same way we think about you know um front end dark patterns and that type of stuff the other thing you you mentioned which is key is diversity in the room and i'll i think one of the probably most uncontroversial examples here well having said that you know um what's uncontroversial these days but um is when twitter started right they had a character limit for user names right right and as it turns out that reflected the types of usernames the people the founding members and early employees um who were predominantly uh white um had but it didn't um um reflect the possibility of a lot of other names around the globe from different different cultures which great examples which are longer right so yeah absolutely uh all i needed was one person with a longer name in the room and uh exactly um we could we could do this this is so much fun man we could do this for so long um and we'll have we'll have to get back to it again at some point but i'd i'd love to just give you um the space i'd love to hear what you're what you're focused on now and what you're excited about now yeah uh definitely want to share this in case uh listeners are interested in this space i want more people to to apply this to um let's call it previously low throughput slash previously ignored industries um so as i mentioned my background is biology and chemistry and um i've you know i've developed some some expertise in in application of machine learning and dealing with data and i want to take it back to that space i've worked in the clinical informatics space but in my head i kind of think about that as uh sensor generated slash human generator like precision on college generated data whereas to me um biology genomics is more like you know pure nature science generated um and it's it's it's really evolved from a low throughput to a high throughput field um in just in the last half a decade and it's continuing to get even bigger and better where you know we have even uh entire like cloud labs where things that were previously you know i used to do uh almost a decade ago things that i would have to do with the pipette to manually observe and you know arduously do over a period of a week are done quicker and automatically where i'm still designing the experiments initially and things are happening and then eventually even the the experiments themselves are being auto designed based on some objective that i'm trying to solve so uh these these kind of waterfalls are being uh kind of unblocked here and specifically what i'm working on right now is just because again not to force ml on everything but because of the modality of the space um you know with drug discovery and small molecule design ml is already being very well leveraged and it just makes sense for the space so yeah i've been kind of uh kind of been heads down on on looking at um applications and you know i like research but i've always been a product guy so i'm trying to figure out uh the specific kind of aspect and problem to solve right now um with some couple folks and there's i'm actually leaning towards some non-profit efforts actually as well because i think while the data bottlenecks are removed there's still a little bit of work that's that's needed before we can start to have full-fledged products and the mrna that's in most of us today most of our listeners i'm sure as well uh is there so quickly because we needed that but all future work that is in that vein needs to be so well thought out and we can talk about risk in this space forever uh because there's there's so many so many downstream implications but uh in my head i'll leave with the summary in the last uh many decades we've been putting code into hardware you know uh silicon uh i think the next the reason i'm so interested in this space and i'm getting deep is i think the next perhaps century will be about putting the same code uh you know concept of code but inside wet wear that could be at the plasmid layer to design proteins to design enzymes to do certain things and i thought so you know this that may sound like scientific science fiction but there's already organized packages to be able to control things at these granularities and it's super exciting it's also uh super scary as well because we need to do this again very uh responsibly and methodologically but of course super interesting to be to be alive even at this period in time is there an argument though that even though that does sound futuristic in a lot of ways that we do do this in some ways with pharmaceuticals and a lot of other things already not quite altering genetic code per se but definitely altering our physiological makeup with stuff made in laboratories oh absolutely um i think the big change that's coming is the process itself uh so initially you know by the way compounds are used everywhere right uh pharma energy material science and so far it's been about leveraging naturally occurring uh molecules and compounds and then certainly the design process to break apart from that uh the build test and design process is super complicated super arduous and this is where ml comes in to help navigate and shorten and kind of shrink that space but yeah it's absolutely happening actually if you you know big examples that people don't immediately tie on to this are uh around synthetic biology like uh beyond meat or possible foods right these are these are companies that are working around the clock to make it cheaper better by engineering things at a more primitive level to make it more x-like um but yeah it's it's all around us and it's it's biology for everything if i'd say that way so it's not the biology for love biology forever and it's also and it's it's actually called tech bio now uh kind of a tech enabled bio so yeah the difference between that and biotech it's kind of flipped um so it's before it used to you know a focus is always on biology by the way that's that's the kind of the foundation of it but now it's about uh how do you develop innovations that really leverage technology to solve uh bottlenecks and things like that that are innate part of innately part of the biology aspect um it's also marketing uh employed by many different companies but um but yeah tech bio is being kind of the adopted term that's great um i love it and i love that you mentioned beyond meat as well because this is a nice example of something that's now deeply in our cultural consciousness and it's being adopted at scale i actually a while ago had um uh like a beyond version burger at burger king oh i didn't know the others there okay yeah so that like they've really taken taken it on um and someone asked me how did it taste and i said it tasted like cardboard and they said well that's a great reflection of burger king so they've done the job correctly yeah i think so um but um that's great goku i am let's wrap up now but i just want to want to thank you for such a wide-ranging thoughtful conversation i appreciate you um and thank you it was so much fun and i i look forward to doing this doing this again oh absolutely uh thanks for having me and i i i'm gonna be definitely tuning in to the next two uh with jakobo and hamel as they kind of understand more details yeah everything that i talked about yeah um but yeah hopefully we get to do this in person sometime too uh yeah that would be very exciting and everyone who's joined today and is watching after the fact thank you so much it was great to have you here um there were some questions you asked i'm going to share this link to our community slack and we'll be there for the next week or so for an ama so just in your own time come to our ama channel there um and goku and i will be responsive but as you can tell we're we love talking about this stuff i mean we could go on for hours more so we we look forward um to seeing you there um so thank you once again goku awesome thank you so much and scene oh