Lukas Beiwald on ML Tools and Experiment Management (2019)

Key Takeaways

I'm Lucas so I I run this company weights and biases that's here and been friends with Josh and surrogate some the other folks for a while and I've been really enjoying all the lectures and I'm enjoying all the projects folks are working on and I really appreciate you all coming here weekly it's it's it's fun to see you and fun to see all this stuff you're working on so I just want to make sure you had my contacted foots Lucas at WB comm so you know if you have any questions or you know you want to talk about any topics I would love to keep the conversation going I want to talk about machine learning tools and it's specifically the experiment management product that I built but I know a lot of you are pretty entrepreneurial and we've had some amazing entrepreneurs come in so I kind of want to talk a little bit about how I thought about building this company and how I have seen the space evolve because you know my my first company was called figure 8 that actually started as CrowdFlower with Chris over there about 12 years ago that collects training data for machine learning and I'm really trying to get you guys credits for for figure eight we actually just got acquired so it's been a like a little tricky to navigate the bureaucracy that I've built just painful painful for me it kind of breaks my heart but if any of you want to collect data on figure-eight I really really want to help you do it on the the platform of it it make me so happy so I have a list of a resume all dress and I'm like begging the team to get you guys account to some credits in them the thing I really can't control is we can definitely get you access to i'm weights and biases which is an experiment tracking platform that i bill is ever used it yet awesome thanks guys awesome i'm and if you need any help at all especially with that please please talk to me i would love to help you out there you know i wanted to talk about some of my favorite AI applications maybe Sergei I could grade them if he has a sorry I snuck off man I wanted to I was kind of thinking like how would I grade these applications you know one super cool application is this microwave it's controversial in this company but this microwave Sergey what do you think I check this out where does this land on your scale so it looks at your food and then it cooks it perfectly supposedly so books and everything yes is that videotape will send it off it is against it cynics business of meta marks like environmental impact or just you know good anyway if you try it he's in the room in there I do think this could still be done right okay here so every talks about self-driving cars but nobody talks about self-driving suitcases there's a digital benefit well actually this is not a benefit at all so does this problem I tried to tried to get one even though you actually probably can't take them you can't carry these onto a plane right but it will follow you onto the plane which I guess so yeah I think this would get higher marks in the car for if it if it fails to follow you right yeah yeah so you could a human operator override it's probably probably easier I also like just imagine the train get is like people's butts it's just kind of like like recognizing you like from behind from like a suitcase saying good and then um this oh man this I think it maybe vaporware breaks my heart if this is vaporware I ordered this about a year ago and so I've not received it but maybe what have you I make this I would be set maybe Stacey might prefer this real viral impact although a question I question the environmental impact but it is super cool so according their video it looks at the trash as you put it in and then decides to put it recycling or not recycling so maybe cool this is actually another this is one that I made recently that I'm really proud of that only works in my demo video also kind of vaporware but I'll sell you one one of these we have one of these so that's like a radiant heater you know and so this is actually it's actually you know space heaters about ten times is efficient as a radiant heater because the ready to here beams the heat at you so sometimes it follows me and I cut the video precisely to make sure that it only shows the part where it follows me oh what's that doesn't feel group Adolf is wonderful because it's like warming you like the Sun it's like amazing I mean my feel different of you but it's like it's like face recognition for my benefit feels amazing for my warmth I don't know anyway I've been really I've been really enjoying your your projects I'm really excited to to see them I think you know one of the things that's really cool right now is there's so many more possible applications then you know people to to make applications that kind of know how to use this technology that I think this is like the best way to democratize the eyes for you you all to go out and make really cool stuff we'd love to love to support that so I want to go through like some of the challenges that I see cuz you know been watching this face for like you know 10 12 years and just seeing the same challenges and they don't seem to go away right and people really work hard to overcome them I think one big issue from like a bureaucratic perspective is that you know as doing machine learning you look a lot like engineers but in fact a lot of things are different right so you know the the code is like incomprehensible it doesn't version well it's like much bigger than sort of something you might check into github and actually your whole workflow is different and I think everybody's had a different take on it sory I'm at a slide and there's almost everybody that um has come in and spoke I think unprompted has actually had some variation of this right where there's clearly like sort of a data piece in the beginning right like a model building piece and then a deployment piece right and I think the your kind of underserved by the different classes that we have is as awesome as there's so much good online material about sort of the learning algorithm piece right and there's just so much less material I think that the promise of you know this applied deep learning class is to sort of help you with the end-to-end process right because if you can't get good data you can't see machine learning right and if you can't to play it and monitor is like what's you know that's just a toy right it's not like a useful thing so this this workflow I think is like fundamentally different and and what this means is that there's like a whole new set of tools needed and the tools I kind of grouped them around the challenges that we face right and so you know the first one is machine learning requires training data and it's something I noticed you know many many years ago like my first jobs just like collecting training they actually Peter Norvig this is like one of the best papers ever I'm not sure nothing is the unreasonable effectiveness of data and it's actually aged like beautifully I'm like you know I'm like some papers would say you know 15 years later not relevant I think this is like even more relevant we're basically Peter Norvig points out that you basically the the the applications that work are applications where there's a lot of training data available he points out at the time is 15 years ago I think that's still I'm really true he also has been like posting on Facebook how ml tools companies are stupid which I just totally to understand and disagree with okay I he's a smart guy so you know you get and you get these effects right this is what me and Chris we're looking at like 12 years ago when I started a company which is that like better algorithms get you know incremental improvements better features are better sort of like data engineering before you feed into the model gets more improvement but it's like really difficult to put a process around but then more data often just gets you huge improvements right and actually cleaner data even less studied can can really get you big improvements and I think you'll feel anyone just like worked on you know mushy letters like nodding their head right but it's like this is actually a surprise I think to everyone when they first go through it because there's not as far as I know a class at you know it's Stanford on collecting data well and and really like if we were optimizing for high quality deployed models they're like half the classes we've begun collecting data well because the other thing that you really can't do is kind of toss it over the fence to someone else to collect the data for you I think um you know Andrey Carpathia like put this best so you know he's the person running machine learning at Tesla trying to make the the autopilot work well and you know he's saying that he spends most of his time on collecting data sets I actually something today needs basic saying I spend all my time on dataset collection right I'm not doing any modeling basically like they know that there's other folks doing it now so so I think this is like you know this is this is I think the the undercover methods allocation was a pretty good allocation in most cases trying to make models work well and it's been really interesting actually to see when I started CrowdFlower reading and figure eight we had basically no competitors or none of our competitors really thought of themselves as AI competitors us they didn't think of themselves as training data collection companies now there's actually a whole bunch of competitors really interesting takes but you should use figure eight in this class for your projects you know another thing is difficulty is like super hard to predict right and and I think that this is like something that also everyone's talked about it's been really interesting to see so I think Josh had some really smart stuff to say on this kind of how to predict difficulty but I think everyone that's talked to you in different ways has kind of looked into this and I you know I I always point to kaggle competitions because it's like data right but you know I did a competition on data that I knew and the first three days he get this improvement going like you know 35% to 50% like feels really good and then like totally flattened out you know over the next the next month or two right and I was thinking like okay if this was if I was managing a team or even if I was doing this project it would be super frustrating right because he'd get like all this momentum right the beginning and then and then it completely flattens out and the effort was actually skyrocketing because if everybody ever do a kegel competition yeah yeah yeah so it's like you know towards the end people get excited because they're like okay we're gonna get that prize you know and and so more and more people were competing and doing really smart things over less and less value right which is like interesting and then you look at like these you know self-driving cars and I think people just can't help themselves but look at a graph like this two miles per disengage and make crazy statements right where you know I guess Josh said this same slide but I just think the slide needs to be like pounded on right I mean this is just you know this is look this is 2015 right this is um this is happening right but it's just such human nature right even people's like seeing the stuff over and over to to get confused by this and then you know people talk about machine learning you know is like unpredictable and opaque and I've had like a lot of different examples of this but I had a personal example recently that really brought this home to me where I am I have that I read a medium like post about machine learning and you know read a lot of post on machine learning and like this is like typical content I write is recommended and then I got this recommendation from medium just short dudes need love too and it's just like man like how did it come up with that like of course I clicked on it sound like I fed the beast but it's just like I really wanted explainable guys like so badly in this moment I still want it oh so you know thing is that it can feel it can feel it can feel it look super non-deterministic right and I think that this is another thing that people working on it really fill where code feels very deterministic generally right you can just rerun the same code twice and you get the same results but then you know people have talked about this machine learning reproducibility crisis it's like such an issue right like you get cut off github and you can run it if you get like machine learning code it's gonna take you a long time to even reproduce one thing that the person did but you know to do all these different things will take so much time and that's why you know we really got excited about you know building weights and biases you know when we were thinking about like even going back for ourselves like one week or one month earlier and trying to figure out what we did it was like you know incredibly painful right and you know my friend actually showing me his notes of the different models that he ran he actually had this Google document that was like thousands of pages of this right where it was like this is the first thing you try this is the second thing you tried and you know I I mean this is like so far from reproducible I just you know I like I like blows my mind that he was thinking he can't like go back to this and and like you know rerun something so it's like what you get from this I think like what's really exciting to me is is you know new workflows and new tools and obviously I'm like incredibly by right but I'm also looking at other tools and I think it's important for you to know what the good tooling out there is right so you know one place that I think you really should be looking practicing in the spaces Siobhan's machine intelligence slide so I think it's just interesting to look at her first one which is because it was very good in a really nice framing of the industry and their second one is kind of getting like you know a little more crowd and then her third one it's like you know getting um you know intensity but you you can her websites evangelist comm is actually just an amazing place to sort of see the entire you know kind of a third party take on the taxonomy of tools and then applications of those tools and I even think the diff of like you know it's kind of 2.0 versus 3.0 is a good place to see kind of where momentum is happening and and kind of where you know interesting applications are taking off and where interesting kind of back-end tools are evolving there's actually another I think just if you're interested tools at all there's a really really smart VC at a red point that I have no affiliation with I just think she's actually writing really excellent stuff right now where she kind of has her own there's a little bit more of like an investor landscape but I think for those of you that are really thinking from an investment perspective or if you're if you're think of pitching VCS on an AI thing in any way I would read her her stuff because it's it's really good but these are sort of the I mean I think the I think these folks are like kind of trying to give you like a complete picture of like everything out there I think what I want to do kind of briefly is zoom out a little bit just sort of give you my kind of quick tuck sonna me would just be very similar I think to to to the stuff that Sergei was saying so one is that you know clearly like there's a point solution versus end-to-end dichotomy and I I myself and I think a lot of people with a little bit more of a hacker mentality I think tend to prefer point solutions so it's really important to me that you know weights and biases builds point solutions because that's what I would want but I do think people in in more regulated industries tend to seem to prefer more antenna platforms because everybody's Domino data any of you yeah but how did you feel but I'm gonna do its [Music] was not just like if I were a new project working by myself oh maybe you were yep but it would be really hard like there's no incremental adoption option so if I was working with a team that had a bunch of like tools right right yeah I think that's the trade-off right so I think that I actually really admire Domino data I think the issue though is you really really have to get everyone to commit and that feels really hard to do I think like even like FB Lerner which is like you know Facebook's tool which is kind of well known for having like broad adoption inside Facebook it's kind of unique in that it's like a company where people really committed to it and actually you know we were over there and it seemed like they were still kind of like rounding up there you know people to to get them to use it so the problem with these tools is that when you don't have complete adoption it can be really difficult to use them I actually at figure eight we used cube flow and it kind of had that same issue right where I was just like if you don't completely buy into it it can be tough so I would not necessarily my take would be for your like kind of projects I would not start with an intent to unless you really want to learn about it because they're all really designed for companies where you get complete buy-in for the the tool and that's where you get the real the real benefit so I think point solutions are going to be better for you I think you know paper space the usual thing sig opt is the you know hyper parameter optimization tool you know pachyderm is a data set versioning tool so I think picking and choosing tools makes a ton of sense and I always am curious about the stack that people are actually using because I think the space is so fluid right now that it it has not converged around it doesn't even converge around frameworks right so I would love to hear what you use and what you think is is useful we actually at figure eight we put out a survey of what people are using every year and see if you go to that website you can actually kind of just see sort of a ranking of figure its customers essentially like you know machine learning practitioners and what you know kind of what has has general adoption you know think like what what I try to think about for my tools I think is kind of lesson sort of the infrastructure and frameworks realm and more in the dev tools I think you know figure eight was kind of a training the other thing I think weights and biases tries to be in the sort of or you know learning algorithm valuation basically it's an experiment tracking thing and it's a that's I think where where I like to I think the infrastructure stuff is really cool and actually much more evolved and settled so I think I think sage makers is a great anybody's sage maker how do you feel about it what's that Oh next yeah I think stage maker I would say of all the I mean they don't pay me but but I think say maker is just it's the probably the one kind of ml infrastructure thing that we probably see the most just generally going into companies right now but of course that can change quickly um and then you see these frameworks like you know clearly tensorflow pythons and carrots are kind of the top most common frameworks that that we encounter so I thought the most fun thing to do really sense I think Sergei gave such a comprehensive overview of all the different tools would be just kind of want to show you the tools that we built and and what they do and actually you know play a little bit with this stuff quickly so if you would open up your laptop and go to op WNBA I slash settings that would be fantastic and you can maybe even do it on your iPad maybe and you know I got this friend this co-founder Chris right over there and he just he loves giving tech support it's like his favorite thing to do so if you have any problems at all like you think you might have typed in the URL wrong this is it Chris got you [Laughter] but then this if you put this code in at that URL what you do have an account then that gives you keep that code on oh yeah I got to keep the code up Simpson stochastic base every day we choose a different code that is a combination of a machine learning word and a musical instrument The Simpsons part tells us to check out this really sick machine learning repo that we set up for you it's a very basic multi-layer perceptron to classify images of susan's characters we're going to brief competition to see who can make the best model we're gonna do hyper perimeter search by crowdsourcing all right so here's the code that you run Python space trained up I will run this little Karras classifier so do you see that so if you go to file and then new and the terminal you get a terminal and this will so this is starting to train my or my this is already trained Chris's little Simpsons character face recognition classifier so the the code that ran is here and so the the first piece is that there's a couple hyper parameters that we set that you may want to modify and then there's downloading and then we make these carrots image data generators to generate examples of Simpsons characters for the both the the training and test data and then right here is the Charis model so if you if you used carrots before and you want to play with this model you could make it a convolutional model or you know something more complicated if you if you haven't done a lot of playing with carrots you can actually change these these hyper parameters here but the goal I guess is that to make a better model I do want to make sure that you all get to keep moving your projects forward so I think we should probably pause this so if you want to keep working on it maybe we'll pull this up in the next class and yeah okay we'll keep it open I mean it I don't know how much we next class when I I mean I would prefer that you worked on your cool projects I would also prefer that someone ooh someone did knock Tom off Christina who are you Christina hey well done well done I guess you teach the next class do you want to say to think about what your approach was interesting one thing just to point out is you actually can go you can go into here and get a sense of of what happened with with this model so if you click on the if you click on this thing on the left here you can kind of dig into to what any particular person who's doing in their run but we'll keep this I mean just for fun we'll keep this runner for a while and if you want to keep yeah so in when you do your when you do this run in the terminal it's going to tell you that it's synced to this URL here [Music] but anyway you can also use this tool for your your personal projects and that was super fun thank you [Applause]