No Priors Ep. 26 | With Weights & Biases CEO Lukas Biewald

Key Takeaways

[Music] we've talked to many practitioners who are pushing the state of the art this week on the podcast we're exploring the dominant ml developer tool weights and biases a lot and I are sitting down with CEO and co-founder Lucas bewald he has a knack for creating companies that support pain points and ml development his first company figurine addressed the problem of data collection for model training and his second company weights and biases has created an experimentation platform that supports AI practitioners at companies including Nvidia openai Microsoft and many more Lucas thanks for doing this welcome to no priors thank you great to be here Lucas you studied at Stanford where I assume you discovered your interest in machine learning and under one of our previous no priors guests Daphne Kohler can you talk about when you start working in Ai and learning from Daphne yeah totally as a kid I was obsessed with playing games and I got really into go and I was super into the idea of or thinking about how would computers win at these games and so I actually sent Daphne an email maybe as a freshman being like hey can I can I work with you like I'm really interested in games I want to learn how to like beat go and Daphne wrote me actually a pretty polite email being like that's not what I do go away a few years later I I took her course and I was actually I studied math at Stanford and I have to say Daphne cared about a thousand times more about teaching than even the best professor in the math department and so it was really just eye-opening like I just loved how much she actually cared about teaching and it got me really excited about the AI that was working there and I went on to be a research assistant for her and it's a funny thing at that time was like nothing really worked like it was just before kind of you know Google was thought to be really like page rank at the time was the thing that was making them work and I think later you know became clear that machine learning was a very a big part of that but really when I was doing ml it was like searching for applications that were working and Daphne was actually really obsessed at the time with a thing called Bayes Nets which you don't hear about too much anymore because I don't think they ever really um you know worked for many applications I hope I'm not offending anyone but that's my my understanding I actually think you know the thing that I really took away from Daphne that that really lasted with me was um I mean she was one of the smartest people I've ever encountered and she had this incredible Clarity of thought and an intolerance for sloppy thinking that that's just like really served me well I think that's sort of separate from machine learning you'd see like other professors would come and give like guest talks and you know they would say something's kind of lazy and like we'd all just be sitting there just like waiting for Daft to like eviscerate them and I think her personality is mellowed a little bit over time but I kind of miss I just missed that sort of like aggressive clear thinking um and I I really admire it I don't think we got a taste of that but we did talk about whether or not probabilistic graphs are are coming back a little bit how did you how did you go from you know Stanford to founding figure eight yeah you know it's funny I actually really struggled doing research with with Daphne basically the things that I tried just barely barely worked like you know I published a couple papers that I feel kind of ashamed of where it was sort of like go from like 68 accuracy to 70 accuracy in a task nobody cares about by throwing like a thousand X to compute and by the way like kind of guessing the most likely answer is probably like 64 accuracy so um you know it just it felt honestly kind of pointless and sad like I love the idea of like computers learning to do things but it's hard to sort of sustain the enthusiasm for that when everything you try just completely you know doesn't work and even the things that do work you kind of wonder if you're like p-value hacking like okay I tried a thousand things you know so I guess something's gonna be like a little bit more accurate than a baseline what tasks were you working on did you did you end up working on go or games or anything no Daphne was Daphne is not interested in games let me tell you and it's actually another I kind of admire that that perspective too as much as I love games I'm a go nerd so I'm curious oh you are oh me too I I yeah I I love go yeah Daphne was very not interested she really was practical and so I worked on a task that you really don't do now called um Word Sense disambiguation where you're trying to find out like okay I have the the word plant actually if you look in most corpuses because they're government generated often at the time plant typically will mean like the power plant sense of plant our cabinet often means the sort of president's cabinet sense of cabinet and so you're kind of trying to figure out like what is the meaning here of these words and then applied it to um to translation it's a cool task I mean and actually it turns out I think that these again nobody kill me but my general sense is that these sort of like linguistic oriented strategies really don't work that well it's kind of like by feeding more data in and sort of like working on outcomes you can figure these things out much better so um a little bit of a dead end and actually in you know I was so frustrated by that that I I just really wanted to work on something that people cared about I actually turned down an offer from Google because they didn't tell me what I would be working on to go to Yahoo because they they were like okay you can work on you know search rank ranking in different languages and but that actually turned out to be incredibly fun right because it was super applied it's actually a task that works really well and Yahoo is kind of in the infancy of switching from hand toon weights to machine learned way so they really had no one not many people actually like working on deploying this stuff so I was like writing code to translate machine learning algorithms into C code and then check it like we would check it into our little code base and run this kind of like semi-hand generated C code in in production so that was that was super fun but you know the thing I learned there actually which I think I'm not the only one that learned this but I just felt it I would go from like country to Country trying to switch from hand tune weights to an ml model and like I was sort of the messenger here so like sometimes it would work and sometimes it wouldn't and so like people are either really happy with me when it did work or they'd be really pissed at me when it when it didn't work but I kind of realized actually the model that I'm building is like the same for each country it's the the training data though is different so some countries would take the training data collection process really seriously and they'd get a great model and some would just like really half-ass it or like you know have these crazy like issues in the data collection and then the model wouldn't work and so I just really kind of viscerally felt how much the the training data process mattered and I kind of felt like you know why don't they let me get involved in the training data process like that would be a better use of my time than building these models and so I wanted to make a company where the people doing the ml could actually have control over the training data collection process and really get like visibility into it because you know at the time I think the thinking was like oh this is sort of like a manual task that's like my like an operations team should deal with this and and they would like they would do this thing where you'd like make this giant requirements document it's gonna sell like waterfall like it would be like yeah it wasn't iterative oh was iterative and it'd be like you'd make like a 50 page document and like you know that the people doing the labeling are not like reading that document but you kind of need that's like cover your ass if they did like labeled something you know not the way you want and it would have been so much better to be like look we're trying to make search results like put yourself in the mindset of like someone you know who's like looking at this like is it good or bad versus trying to lay out in like excruciating detail what makes something relevant or or not relevant I think also at this time like when when you first started um I think originally was called Dolores labs and then crowd flower and then eventually figure eight like I think I met you in your Dolores Labs days or something I know I remember yeah yeah yeah and at the time there weren't really um solutions for data labeling externally right some people are using mechanical twerk from Amazon to sort of run jobs on untrained workers there wasn't like scale there wasn't you know there was none of these Services yeah and so you got really early to this idea of starting like a data labeling company and that that was actually very useful for machine learning and so it'd be great to hear like you know what were the early days of of that like and what was it industry like and how did you get all that running yeah I mean it was funny right because back then I was coached actually quite a lot by You Know Travis kalanick who's you know famous now for for doing Uber and other things but he was like don't tell anyone that it's like AI like VCS like don't want to hear Hey I was actually good advice um at the time and it's good advice in the early days of the company and started interrupt I think one interesting side note on that just from a Silicon Valley history perspective is Travis used to have these effectively like hackathons or meetups at his house called the hack pad and you know I think you used to go to those you know a bunch of friends of mine used to and so a lot of startups actually had some impact or influence from Travis in those days like due to his fact of like you know being another founder in the scene and kind of getting everybody together and so it's kind of an interesting moment in timer and history and to your point back then like AI wasn't really as popular as it as it became later so it's kind of an interesting like side note well I mean Not only was a AI not popular but like startups weren't popular right like my family didn't you know understand about startups and I had graduated Stanford you think I'd have all these great like connections but it didn't feel like that like I had no one who knew how to like raise money from feces I didn't know any you know VCS or I didn't really know any like entrepreneurs honestly and we had this website for Dolores labs and early days just trying to get customers and it put my my personal phone number actually remember I was like the first user of twilio because I needed to make a phone tree and so I used twilio software and then like all three of the founders came to my house to like help me like make that phone she like work better which is kind of amazing it was like you know like you know one of those like you know 20-something like um you know grunge apartments in the mission and then uh and then Travis called in but you know it's funny because the phone treat we were just trying to pretend like we were a big company and Travis called in because in the phone numbers on the website not because he wants to buy anything but he just like thought it was like awesome and so I'm just like you know I pick up my phone and then there's just like this guy and then just be like oh man like this is so cool you know I'm like okay like who are you you know it's like it's like hey do you want to like a coffee and uh and that actually turned out to be incredibly like helpful but then I I think like the thing that was so different back then is that the people doing ml there just weren't that many like there were people like heavily investing in ml but they're but it wasn't that many and so what happened was you know we got like eBay as a customer which has really mattered at the time and we got like you know Google as a customer and Bloomberg and then there just like wasn't anywhere else to go so like you know my board was always like recommending like read crossing the chasm and and we tried like a million different ways to like you know grow the company and you know I don't know I hope this doesn't sound defensive I mean maybe I'm just a bad CEO but we had like years of like struggle because there was no Chasm to cross right there was like nowhere else to go so we tried all these different things to like you know build more complete solutions for our customers and it just didn't work and then kind of all of a sudden um you know autonomous vehicles got popular and that really actually suddenly caused our Revenue to um you know start to to grow really fast again but it was like an eight-year lull of like you know really no growth right so it's hard because we started a fast got everyone really excited you know kind of got like whomped for just like years and years and years actually we had all these competitors they all went away so at some point we had like no competitors left right because like everyone had uh had gone out of business and then it was a funny experience because like scale came along and totally ate our lunch on in the self-driving mark which is a market like I knew and loved and so you know I I was so excited to sell the company after you know so many years of struggle you know but then like right after that we see like scale just like skyrocketing and revenue it's like oh man like I wish we had just like you know maybe held on a little bit longer but then you know it gave me the space to start weights and biases so you know who knows I I want to be like Daphne color and evaluate my decisions like accurately and and critically but it also does seem like you know I've had some good luck along the way yeah I know the market shifted so dramatically and I think to your point self-driving was the first time that you suddenly had a bunch of systems at scale that people needed data labeling for and then of course now we have this llm way but it's all very very recent and I think a lot of people basically view ml as this sort of continuity and everything has always been kind of rising in a sort of almost linear way and in reality it's this very bumpy set of discontinuities in terms of the set of Technologies and markets that people are adopting it in and so it's not continuous it's a discontinuous thing and nobody thinks about it that way when you started weights and biases you said something along the lines of you can't paint well with the crappy paintbrush you can't write code well in a crappy IDE and you can't build and deploy great learning models with the tools we have now I can't think of a more any important more important goal than changing that and that's I think like when you announced that you were starting with some biases and so I was just curious like what lapses and capability really got you going on um 1B and can you also just you know many of our listeners um know what it does but for those who don't could you explain what the product does and how it works sure yeah so it's kind of constantly evolving right because we're saying it's like a set of tools for for people doing machine learning we're best known for our first thing that does experiment tracking which keeps track of like how your models like perform over time as they learn and train oh we also have a lot of stuff around like kind of data versioning data lineage you know production monitoring model registry kind of the the sort of end-to-end stuff that you need to do machine learning reliably and I think the thing that happened to me was I had been running cauliflower for years and I always loved machine learning but I was like really starting to get out of date like deep learning came along and at first I was kind of skeptical of it because people are always saying I have a better model that's like magically better and they're like wrong wrong wrong wrong wrong it's just like really like data and then but then they were right right so there actually was a sort of a better modeling approach that worked and I kind of realized you know when I was in my early 20s I was really judgmental of you know the people in their late 30s that hadn't like adapted to machine learning at the time because like rule-based systems were kind of all the rage when a different generation was was growing up and I was like wow you know I am actually getting out of date myself like I'm saying these kind of wrong things that were true 10 years ago and are not true now and I honestly felt like really bad about myself and so I did a couple projects to try to you know get up to speed I started teaching free machine learning classes and and deep learning classes to kind of force myself to to learn the material and actually like interned briefly at um openai where I was just like look I will just do whatever you know work you want just I want to be like I need I know that I need like an accountability partner essentially to force me to learn stuff even though I love to learn stuff like my favorite thing but I always need accountability practice for anything I do so I sort of use the students as an accountability partner and open Ai and then what was happening was I was showing my old co-founder Chris like all the the cool stuff and he's like a really good engineer and I'm like actually really like bad engineer like I'm like really lazy and I try to write the like you know I'm just like like people my co-founders make fun of me all the time for like you don't really know how get works and I just openly I have no idea how git works I just sort of mash the keyboard until like I kind of like you know get an event date and then I like call Chris and beg him to like yeah [Laughter] I don't understand it and it's like my co-founders just find it like baffling that I wouldn't understand it but I think it's like um for them you know it's like they're like wow this guy like needs some basic tools you know like because you know they're like okay like reproducibility like why don't you just use Docker I think that's sort of the Ops mindset but I'm like man I don't understand Docker guys I feel like I installed on my like laptop and then it's always like taking up memory and stuff I like I don't like don't really know what it's doing and I'm like kind of scared of it and like I don't know so it's like I just feel like it's adding weird complexity to understand and so I think the tools kind of exist in a way but they just weren't made in a way that like ml people could really use them because like you know if you're like me you kind of come from a mathy background or like a research background you kind of didn't really learn to do like industrial style coding and so you know I think companies have this idea that like the researchers are just gonna like throw the thing over the fence and then it's going to be in production but it doesn't really work actually like I think that's a bad pattern that people sort of like imagine they're gonna do and they don't ever really do that you end up like with research is always the research code bleeds into production and every company and so I think a better way is to give you know researchers and like ml people tools to just make their stuff more reliable and it has to be simpler maybe or it's just a slightly different audience like you can't just give someone like Docker you can't just like you could I mean a lot of people like hey why don't you use like the get large file system stuff to to version your data and like there actually are some reasons like it doesn't work well with like object stores so there's some like ergonomics reasons but it's also just like man git is like complicated I'm like willing to use it for code but if you start making me like version my data with Git like I just want to like cry you know what I mean so like give me something like simple you know what I mean where I don't have to like think about it or I'm just gonna start like renaming my data sets like latest latest really latest session relief for sure June 27th so I I just need my stuff to be simple that's kind of the mindset you know behind the companies like let's like make these like kind of simple clear things that actually help people we were talking about how much you wanted to like you were thinking through how much LMS were gonna change like experimentation and ml tooling when we last saw each other in person not at the zoo but before that yeah and you you guys launched this prompt Suite in April like can you talk us through the sort of you know thought process of hey like you know I I really admire this as a leader and as a technical person you're like trying to stay really plastic about what is actually changing in machine learning how do you think through this change well it's really hard right I mean so what happened was we have a great business that you know makes like an ml uh a set of ml tools for training models and we actually helped most of the llms out there were built using weights and biases and then we started to see like wait a second some of these ml tasks you could just ask the llm right so instead of doing like a sentiment analysis model you could just be like hey like is this document positive or negative sentiment like for structuring documents you can just be like hey find all the names like in this document and it actually works super well and a little piece of me is a little bit sad about that because we have this like great simple relaxing business that grows Revenue every every month that I always dreamed of right so you know part of me is like [ __ ] this is actually our kind of first real existential threat I think you know and and um and you know I went to my like leadership team and I went to my board and I was like I think there's like a real existential threat here and I think they were like hey you know we don't like see it in the data like are you sure like maybe you're being paranoid and I guess I do feel sure and I don't want to say I'm like the only one or like pay myself as the hero like you know my co-founder is also seeing this and you know people talking about it but it's sort of like you know this threat is like now right and we have to actually like get the whole company to to do this thing because it doesn't show up in any of our like metrics yet but I just really believe that you know our customers are rational and they're gonna do a thing that like makes sense for them and so I see a lot of my colleagues being like Oh there's going to be like lots of different models it's like nice if it were true but like what I see everyone doing right now on July 27th is using GPT like I didn't see like 95 of the people out there you know using GPD for these ml tests and so it's like look we gotta support that and so we've really rallied the whole company behind it and uh we pushed out prompts we'd also this is really my my co-founders my co-founder Sean had really put a lot of effort into making our stuff really flexible because he's like you know what Lucas like there's gonna be like changes you know coming we don't know exactly what they are but like you know kind of from the beginning we really tried to build very flexible infrastructure so this was kind of a moment we could really sort of like Flex that and get out of um you know a product for for monitoring stuff and you know now it's like you know kind of it's our number one priority is getting out more tools for this new this new workflow out of curiosity because you know there's a lot of debate right now in terms of proprietary models versus open source models and um I think there's a really great quote I think it's from Harrison from link chain which is you know no GPU until product Market fit right you should first like figure out if the thing works at all or if there's a customer need and that means using GPT and then once you prove it out you know you may use gpt4 or something for very Advanced use cases and then you kind of fall back to 3.5 or you start training your own model for things where you just want cheap sort of high throughput things happening and it increasingly feels to me like people the most sophisticated people who are at the farthest sort of Cutting Edge on this stuff are kind of doing both right they they use GPT to prototype and then in some cases they're they're training their own incidence of llama 2 or whatever they're using do you think that's where the world is heading or do you really think things kind of collapse onto some of these proprietary models like over time like it's six months from now it's a year from now it's two years from now I'm just sort of curious about how you think about adoption of Open Source it was funny I feel like lately what I've been telling people is like I'm just trying to see the world clearly as it is today I can't predict the future and I can barely keep track of you know what people are doing today when I consider it like my my full-time job so I'm like scared to prognosticate like what you know might be coming but I think you're right that that's what's happening now I think like there are like a bunch of things that could change right like I think like you know GPT is way far out ahead and it's hard to fine tune it not even possible with with gpt4 and I think that that is like a little that's not like a technical limitation I guess sort of like a business model um you know limitation so that might change I think that there's a lot of hidden costs to running your own model I think people are really enamored with the idea of running their own model and I've kind of seen this before where I think at the end people do rational things but it kind of takes them a while so I'd rather sort of support what looks like the rational workflow I mean I think the insane thing must be crazier to be an investor in this world is like very very few people have llms in production like there's probably more companies that have raised money as like llm tools than companies that have LMS in production which is like insane it's just like an insanely saturated tools Market with very few people getting things out but it's because when you Lucas when you say when you say Ellen's in production you mean my own that I have fine-tuned about it I serve myself no sorry I mean like G like GPT like using GPT in production oh really okay look I mean you you may be quite closer to this to me but it's a small handful yeah I'm like desperately trying to find them because like these are our customers like we you know our stuff is just like our ethos is like we want to help people do things in production so it's like if you're not in production we're not relevant to you so I like I mean back in January February this year we were looking for design partners that had stuff in production and boy was it hard to find right like you know now there are more but even when you you know you find people that are sort of like claiming to have this things in production it's sort of like well it's like you know it's coming like you know we have like all these like sort of like prototypes you know running and so I think it'll change I think it's changing quickly but I think it's a it's a funny moment where I mean I think if you actually looked at the Tam today of like tooling for like oh I was like I don't know I bet you it's um it's small and I think also I think VCS Maybe sometimes have this this funny window where you see like all the companies that are using LMS but the Enterprise adoption has been slower I mean despite the fact they talk about it like constantly like constantly like everyone's talking about it but in Enterprises like boy I don't know if I've like used the product of like any Enterprise that actually like was backed by a um an LM and there's a bunch of things that make it hard it's like you know it's kind of unfair because this stuff has only been out for like six months or so but it is like I think the adoption maybe maybe take a little longer the short term that people think I think that's a really key point because ultimately you know Chachi PT came out eight months ago and that was kind of the starting gun for all the stuff in my opinion and then gpt4 came out in March or something right which is three four months ago and if you look at Enterprise planning cycles for large Enterprises it takes them six months to plan something right and so people often ping me and ask about adoption of these sorts of things and it's like well notion is seeing you know has adopted it in addition ways already a zapier is adopted in interesting ways but it's basically these technical founder-led companies that jumped on it really early relative to everybody else and the big Enterprises are going to take another year or two because it's they're just in their planning cycle still around this stuff they just started really thinking about it and how to incorporate it and what to use it for and then they're gonna have to prototype and experiment for a while and then they'll push it into production and so that's why I was kind of asking a little bit about the future I just feel like it's so early yeah and we all talk about it again as if it's this continuous industry cycle but it's really not it's a disruptive new technology and so you know I think a lot of it's still to come in really interesting ways oh totally and there's tons of product issues too right like you know like notion and zapier both have these really compelling demos and they're both products that I use but then I actually don't use the LM like piece of them myself and I wonder I have no Insider knowledge of the level of adoption but I think they're I think they haven't gotten it like perfectly right yet despite like a lot of thinking and really smart people working on it sure for the core 1B product you know you folks are being used for a wide variety of areas around autonomous vehicles Financial Services scientific research media and entertainment is there any industry in particular that you think you're either surprised by adoption of the product or you're really excited to see sort of how people are using it yeah I mean the one that stands out for me because this is the one that's really different than you know my figure eight days is Pharma so I actually think this is kind of flying under the radar a little bit but every Pharma company is making major investments in in ml and not just on this sort of like I mean they do have these operations to sort of like sell more you know drugs to to doctors that uses sort of like light ml but I think the thing that's really exciting is like the actual testing of drugs you know before they they have to test them the physical world and that's like obviously working you know super well and I think I I see this before too it's like autonomous vehicles and stuff it's like there's a big lag there right before you get something through like all the clinical trials so no drug developed by ml has gone through clinical trials but if you look at the behavior of all of the big Pharma companies I can tell that it's working because they're hiring hundreds of people right like you know like companies will hire like a few people for like an experiment but they're all gearing up to like operationalize this stuff and that just gets me really excited I mean they could all be wrong I suppose and I don't really have any Insider knowledge except for the seats that get bought on you know wasted biases but when I see that I get pumped because I I just like you know the drugs that they're working on you know the diseases that they're curing it's like the ones that like you know like our relatives have right like you know Alzheimer's and Parkinson's and these kind of horrible things and I think there's just a huge promise in being able to do physics like inside a computer versus in the world yeah I think there's a I think that this is a really important Point too it's actually commonly said like no no machine learning developed drug has actually come to market today but it's a backwards looking metric in a very slow industry right like the clinical trial cycle is very long and and so um I'm actually like quite quite optimistic on this yeah and I don't think that's a that stands out in Pharma because it's very under discussed but there's certain Venture funds that have done incredibly well financially and Pharma where there's one in particular I can think of that never ship the drug until the covered era and they were in business for 20 years wow and they made all this money and they found out all these companies and none of their biotechs ever launched anything in the market wow so I think that's a that's a broader sort of issue with Pharma and we can talk about that I think some other time but it's kind of interesting how how little biotech has actually delivered and there's been amazing deliveries right in terms of different drugs and things but it's actually more common than just the ml side I think yeah Lucas you okay so Pharma is something you're excited about and you think has promise and and growth in um at least seats of 1B figure eight like you talked about you know Yahoo eBay like it's a very small set of people who else do you see in the weights and biases like customer base now like how has that changed since it's it's actually incredible to me that you've been you know working on this from the entrepreneurial side since 2007 because it's like you know pre pre-even deep learning Revolution right and so I imagine you know you've got a much broader user set now oh yeah it's so cool I mean the coolest thing about running weights and biases is the customer set is everyone I I really think every Fortune 500 company is doing something with ML that they like actually really care about and it's always surprising right like we work with you know most of the big game companies like I'm not a big gamer so like I you know like I'm vaguely aware of like Riot games and like unity and stuff but you know but they do all this cool stuff with ml to like you know make the games more fun to make like you know models in the games and this is like big Investments they really really care about because you know again like we're sort of the last step in your journey is to want good tooling for your ml team you kind of need something to work so you hire an ml team you get into production then you like run to problems then you come to waste devices so like we see stuff you know after it works and like you know like AG Tech like you know big agricultural companies I like never heard of some of them when they showed up and then they're like these huge you know businesses that are actually using ml to find ways to do like cleaner farming like a lot of the reasons you know you you spray a whole field with with pesticides it's just cause it's like so expensive to do something smarter and so you know I think I think that like crop yields and the you know the the cleanness of the the farming practice is about to like dramatically um improve like you know we worked with John Deere for years back from a figure eight days to you know weights and biases and they're they've deployed sprayers that only target the weeds in in fields it's deployed it's like you know I remember like for years seeing pictures on the wall and then showing me like prototypes and then one day they're like yeah you can like buy this you know and it's it's cool because like this intelligent stuff it's like software right so it's like it's not like a machine you just like press copy and then you have you know more of it and so so yeah we see that we see like a lot of um you know I mean fintech probably obvious to you guys but like they're kind of I think always on the Forefront you know the stuff for Lots I mean like there's like consumer oriented stuff that you'd recognize like you know making chatbots not annoying right and then there's like you know kind of more you know Financial forecasting and and things like that but yeah I mean it's funny we don't do any vertical based marketing because there's not one vertical that's like dominant enough to to Warrant it and our customers bounce around between verticals so much that I think the Common Thread here is people doing like ML and data science versus any particular application which I just do is super cool that means it's sort of like table Stakes you know for everyone you you know made jokes I think jokes about like not being a terribly good engineer and now the weights and biases messaging is very much about developer first right can you talk a little bit about how you think about like you know and it actually it is like yeah as far as I understand it's like one of the most broadly adopted tools tools by developers work at nml how do you think about like developer adoption versus like researcher adoption and what did you do that worked yeah I mean it's like developers and researchers they kind of blend together but I think that I think that what happened in the sort of ml app space is that you got a lot of well the early companies had to sell to Executives which I totally understand like that's what crowdflower had to do and the the problem there is you kind of get stuck in these like multi-million dollar deals and like you just can't get out of that like you can't switch to like a plg motion and so the early companies I think are kind of stuck right with like these products that like cios love and the you know Engineers hate and that's just like I just didn't want to do that with weights and biases no matter how big the market is or how like juicy that is and the good news is it's like not a good market like a developer oriented sales better when you when you look at like developers versus ml researchers that line is really blurred in the time that we've been doing it and and I think that like there's sort of like subtle differences but you know when Nvidia came along and these chips worked for deep learning it just like broke the entire stack like it was like a first time that in in my career where I'm like running into like like Linker errors and what the [ __ ] is a Linker like I vaguely like remember this you know from you know like a CS class I took you know like and um and so it's like I think that ml research has really had to be kind of become software developers and then at the same time you know the AI class is the most popular class so like all these software developers the Smart Ones kind of become ml researchers so I think that line has weirdly blurred but then I I think there's a funny thing that also has been happening where like every devops person on the planet rebranded themselves as like an ml apps person all of a sudden and so you get all these companies that come out of like every ml Ops Team then realizes they could raise like a shitload of funding you know and so like you got like every every major company their ml apps team like went off and like raised money to like make a new product in the market which I think from an investor that's logical right it's probably they have a good thing but they're just like not good at connecting with actual developers right because they're actually like devops is is like a little bit of a different discipline where you're sort of obsessed with reliability kubernetes seems like simple to you and that's just not like the experience of like an ordinary yeah developer like you know like like my co-founders or or me and so I think the joy of weights and biases is we're kind of making software for like ourselves and I think it turned out that like maybe in the median of my three co-founders was actually the the target audience for us here I think I skew more towards you know an ml researcher barely but you know if I had to like pick one end of that spectrum and you know my co-founder Chris probably excuse more thread software developer and Sean's probably somewhere in between one of the things that's common to people or to developers is that they love to write their own tools and they tend to really enjoy using open source over closed Source Solutions how did you think about the open versus closed Source approach and how did you think about you know making something that's valuable enough and good enough to overcome that natural inclination to just do it yourself what's funny like I think the tools thing I've always felt like I've always felt like kind of proud of making tools for developers like that's always felt like really good because I think developers sort of know what quality is like I mean it's like I kind of like making a tool for someone that could make the tool themselves because it kind of raises the bar and this definitely my grandfather was like a pattern maker which is like a sort of you know like the person makes a pattern for other machinists and he had the same attitude of like look I'm making this stuff for like other engineers and like there's like an honor in that so I definitely feel that pressure and love it the open source was closer thing was really just like we didn't know how to make an open source business so so like we kind of started off close to us because we just we actually wanted to have like a working business and it's had like a pro there's been a major probe which is that all our competitors are open source and what that means is that they don't get to see how users actually use their software and so I think our software is a lot more ergonomic because we have like metrics on what people actually click on if people aren't click on a button we remove it if people like you know pick an option all the time then we know to like make that the standard option as we've grown and you kind of can't just like rely on anecdotal user feedback that I think has made our product like a lot better like people find it like nicer to use at the same time I understand why people want to go to open source stuff but honestly I feel like it's a little bit of a devops mindset also like I mean devops people like they're obsessed with like you know open source and usually like the ml Ops people we talk to in companies really want like an open source piece which is why our client is open source everything actually runs in your servers is open source but like I don't know like ml researchers aren't so precious in my experience generally they kind of want to get a job done and I think they're kind of happy to like that we have like a stable like business that generates money in like a normal way and isn't going anywhere or at least that's what I tell myself I think this is like the part about like the need for like ongoing Telemetry and application feedback like there are a you know zero to Marginal number of Open Source applications that have actually succeeded I think part of it is like this from you know hierarchy of honor of like the deeper in the stack you go like do people really want to work on like web UI in the open source or just like random business Logic on a relational database like yeah it's not as sexy and exciting to like go put your like GitHub badge on but I think the piece that you describe is actually really important where you know you work on complex workflows and if it's something that like somebody can just run in infrastructure and it's like you know you you get data back on like config files or yaml or whatever like that might that might work in terms of like one person's architectural point of view or some framework but I really don't think it works at the application layer for for these two reasons right like one total lack of feedback and to sort of the lack of interest in the I don't know technical brownie points you get for it yeah do you still pay attention I'm sure you do actually to like annotation like what do you what do you think happens to the data data annotation space and like you know the land of LMS and ra CHF and such you know I'll be like honest actually I guess I'll be like totally honest I find it like incredibly stressful because I still feel bad that we lost the scale like it's still like it's just like lingered with me and I I admire skill actually I know hard that that businesses so I have just like deep admiration for their like execution but as a competitive guy I kind of can't get over it so I'm like always inundated with questions from VCS like whenever any annotation company's raising I know about it because everyone like calls me but I I honestly try I know I should be closer to it but I try to stay away from it just because it caused me so much anxiety to look at what's going on that I uh I just can't deal with it what were some of the things you did differently with the second company I feel like you know I've started two companies in with a second one there's all sorts of lessons I applied immediately where there are two or three key takeaways that when you start awaits and biases made the second time around easier or was it harder how did you think about you know key key learnings or how to apply new things yeah I mean I think like one thing was like extreme clarity about who we were serving so I'm surprised I don't hear this more because like the the wasted biases started with a with a customer profile and I think it's actually a nice way to start a company because you know especially as like a Founder you have to spend so much time with your customers you have to seek them out like picking a customer that you love I think is a really good thing for your like mental health you know and so that was like a big thing and then I think like I think I've just been a more confident person in myself like any time I start thinking like okay like long term or short-term it's just like you always want to think long term like everybody wants you to think short term like everyone's going to push you to think short-term they wouldn't say it like that but it's like you know it's like people can see like ARR growth they can see like user growth think it's harder to see like product quality right and so I think like I think I'm a competitive guy who likes you know metrics and likes accountability but I actually think that can get counterproductive for me where you know you start like sacrificing short-term things to grow these external facing metrics and I just really try to fight that myself I know everybody like chases every entrepreneur chases like short-term like ARR numbers like in quarter but then it like hurts your growth rate the next quarter it's like it would actually be better always to like push out deals but like nobody thinks like that right you can't think like that but it's I don't think it's totally rational is there any advice that you would give to Founders who are running their first AI company or just getting up and running yeah you know the advice I always give is like it's like the generic advice that everyone says it's like even truer than you think it's even truer than like I know even though I like deeply believe it so it's like caring about like if you're making something people want like everybody knows it but like no one cares about enough right like people just they get distracted they do other weird stuff even I do it I understand but like you should care more than you think no matter how much you think I've never met anyone that cared too much about that and then spending time with customers it's like it's so critical everyone says they do it but I don't really believe it like I feel like I'm obsessed with this I mean like getting like when you're an early company getting like three customer calls in a week that's like tough man I mean you gotta like scrape and Claw and like beg to get those meetings and you know like two of them are gonna like cancel so I know people tell me oh I met with like 30 customers this week or something it's like really did she like I don't know I I enjoy that really hard to get customers attention like so I don't know I have a feeling that nobody does enough of that but I don't really know I think people are lying to each other but how much like actual kind of customer meetings they're doing and then it's like you know when you get to a customer it's so precious it's just like man like show up prepared and like ask the tough questions like I think like I feel like one thing about me is like I always like default to like wanting people to like me and it's a terrible trait in a in a CEO you know it's like I feel like I have all these like coping mechanisms for myself to like not just like kind of flip into that mode but I think it's good for customer Discovery because I'm always like so afraid that they secretly like hate my product you know that I get like really insecure I'm just like okay like you know tell me like more you know like like are you sure this is really like working for you actually because it does actually help in that one important like entrepreneurial process to lean into your insecurities with your with your early customers Lucas this has been great is there anything you want to talk about that we didn't cover no this has been fun I mean I just I think the message that I'm trying to tell the world is that we're really trying to make tools for this new llm workflow that people are calling lmops and some of my my advertisement for weights and biases is like hey if you knew us and liked us for our ml op stuff try our llm up stuff called prompt I think it's I think it's not amazing yet but I think it's kind of ahead of the market and it's about to get a lot bett