Beating GPT-4 with Open Source Models - with Michael Royzen of Phind

Key Takeaways

[Music] hey everyone welcome to the laden space podcast this is alesio partner and CTO of residents and deible partners and I'm joined by my co-host swix founder of small AI hey and today we have in the studio Michael royen from findes welcome thank you so much it's great to be here yeah we are recording this in a surprisingly hot October in San Francisco and uh I mean uh sometimes the studio works but the Blue Angels are right now so sorry about the noise I don't think they can hear it we have enough damping anyway so um so welcome uh I've seen find blow up this year mostly I think since your your launch in Feb and V2 and then uh your your um Hecker news posts um I we tend to like to introduce our guests but then obviously you can fill in the blanks with the origin story um so you actually were a high school entrepreneur you started smart lenss uh which is a computer vision startup uh 2017 that's right yeah so um I remember when like tensorflow came out and like people started talking about obviously at the time the um after Alex net the Deep learning Revolution was already you know in flow and um good computer vision models were a thing and what really made me interested in deep learning was um I um got invited to go to Apple's WWDC conference as a student scholar um because I was really into making iOS apps at the time um and so I go there and I go to this talk where they are um they add an API that let people run computer vision models um on the device using far more efficient like GPU Primitives um and after seeing that I was like oh this is cool uh this is going to like have like a big explosion of you know different like computer vision models running like locally on the iPhone um and so I had this crazy idea um where it was like What if I could just make this model that could recognize just about anything and have it run on the device and that was the Genesis for what eventually became smart lens um I took um this dat set called imag net 22k so most people when they think of image think of imag net 1K but the the full imag net actually has I think 22,000 different categories yeah so I took that um filtered it pre-processed it um and then did a massive fine-tune on um Inception V3 which was I think the state-of-the-art uh deep convolutional computer vision model at the time um and to my surprise it actually worked like insanely well I had no idea what would happen if I it like I give a single model um I think it ended up being 177,000 categories approximately that I collapsed them into um and actually ended up working so well it worked so well that um it actually worked better than Google Lens um which released its V1 around the same time um and so and on top of this the model ran on the device so it didn't need an internet connection um a big part of the issue with Google Lens at the time was that uh connections were slower you know 4G was around but like it wasn't nearly as fast and so there was a noticeable lag having to upload an image to a server and get it back um but just processing it locally even on the iPhones of the day and 2017 much faster um and so um it was a cool little project uh it got some traction techren wrote about it and there was kind of like one big you know spike in usage and then over time it tapered off um but people still pay for it which is wild that's awesome oh it's like a monthly or annual subscription yeah it's like a monthly monthly subscription even though you don't actually have any servers even though we don't have any servers that's right I was in high school I want to make a little bit of money I was like yeah that's awesome the the moding equivalence kind of be my eyes and uh they actually disclosed in the gp4 vision System card recently that the usage was surprisingly like not that frequent like the extent to which like all three of us have a sense of sight I would think that if I lost my sense of sight I would use B eyes all the time the average usage of BM eyes per day is 1.5 times exactly and I was I was thinking about this as well where um I was also look looking into image captioning where like you give an a model an image and then it tells you what's in the image but it turns out that what people want is the exact opposite people want to give you a description well people want to give a description of an image and then have the AI generate the image so exactly and so you know at the time um there I think there were some Gans Nvidia was working on this like back in 2019 2020 they had some like impressive like I think face scans where they had um this model that would produce these really high quality portraits um but it wasn't able to take a natural language description the way mid choury or Dolly 3 can and um just generate you an image with like exactly what you described in it awesome uh and how'd that get into NLP I released the smart lens app and that was around the time I was a senior in high school I was applying to college um College rolls around I'm still sort of working on updating the app in college um but you know I start thinking like hey like what if I make like an Enterprise version of this as well um at the time there was clarify that provided some computer vision apis um but I thought you know this massive classification model like works so well and it's so small and so fast you know might as well like build an Enterprise product and I didn't even like talk to users or do any of those things that you're supposed to do I was just mainly interested in like just like building a type of backend I've never built before so I was mainly just doing it for myself just to learn um and so I build this Enterprise classification product and as part of it um I'm also bu building a like invoice processing product um were like using some of the aspects that I built previously although obviously it's like very different from classification um I wanted to be able to just extract a bunch of structured data from an unstructured invoice um through our API um and that's what led me to hugging face for the first time because that involves some natural language components um and so I go to huging face and with um various encoder models that were around at the time I think uh I used I used the standard Bert and also long forer um which came out around the same time um and long form was interesting because it allowed it had a much bigger context window than those models at the time like Bert all of like the first gen and coder only models they only had a um a context window of 512 tokens and it's fixed there's none of this like Alibi or rope that we have now you know where we can basically massage it to be longer it was they're fixed 52 absolute encodings and so long forer at the time was the only way that you can fit say like a sequence length or ask a question about like 4,000 tokens worth of text um and so um implemented long forer it worked super well um but like nobody really kind of used the Enterprise product um and that's and that's kind of what I expected because at the end of the day um it was Co I was building this kind of mostly for me mostly just kind of to learn um and so nobody really used it and my heart wasn't in it and like I kind of just like shelved it but a little later I went back to hugging face and I saw this demo that they had and this is in the summer of 2020 they had this demo um made by this researcher yes J NE and uh he called it um long form question answering um and basically it was this self-contained notebook demo where um you can ask a model uh a question the way that we do now with tgpt um it would do a lookup into some database um and it would give you an answer and it absolutely blew my mind um the demo itself it used I think Bart as the model and in the notebook it had support for both um an elastic search um index of Wikipedia as well as a dense index um powered by Facebook's face Vice I think that's how you pronounce it um it had both and it was very iffy but when it worked I think the the question in the demo was why are all Boats White when it worked it blew my mind um that instead of doing this few shot thing like people were doing with gpt3 at the time which is all the rage you could just ask a model of question um provide no extra context and it would know what to do and just give you the answer um it blew my mind to such an extent that I couldn't stop thinking about that um and I started thinking about ways to make it better um I tried uh training or doing the fine-tune with the uh a larger Bart model um and this Bart model yeah it was fine-tuned on this Reddit data set called um Eli 5 so basically subreddit yeah the subreddit yeah uh someone had scraped I think I I forget who did it but someone had scraped the subreddit um and put it into like a well formatted relatively clean data set of like human questions and human answers so we bootstrapping this model from from Eli 5 um and that made it pretty good at at least getting the right format um when doing this rag retrieval from these databases and then generating the final answer and so Eli 5 actually turned out to be a good data set uh for training these types of question answering models because um the question questions written by human the aners written by human um and at least helps the model get the format right even if you know the model is still very small and it can't really think super well at least it gets the format right um and so it ends up acting as kind of a glorified summarization model where if it's fed in high quality context from the retrieval system it's able to have a reasonably high quality output and so once I made the model as big as I can just fine tuning on BART large um I started looking for ways to um improve the index so in the demo in the notebook uh it was um there were instructions for how to make an elastic search index just for Wikipedia and I was like why not do all of common craw so I downloaded common craw and thankfully I had like 10 or $15,000 worth of AWS credits left over from the smartland project um that's what really allowed me to do this because like there's no other funding I was still in college you know um not a lot of money um and so I was able to spin up a bunch of instances and just process all of common craw which is massive so it's roughly like um it's terabytes of text um and so I whitelisted I went to Alexa um to get like the top thousand websites or 10,000 websites in the world then filtered only by those websites um and then index those websites because the the web pages were already included in dump so I just you mean to supplement common crawl or to filter common crawl filter common crawl okay yeah so we filtered common crawl filtered common crawl just by uh yeah the top I think 10,000 uh just to limit this because obviously like there's this massive longtail of small sites that are really cool actually and there's there's other projects like um shout out to margin Nan new um which is a search engine specialized on the longtail I think they actually exclude like the top 10,000 I've seen them around I just don't really know what their pitch is yeah yeah yeah so so they they exclude all the top stuff so the longtail is cool but for this that was kind of out of the question and that was most of the data anyway so we removed that um and then I indexed the remaining approximately 350 million web pages through um elastic search um so I built this index uh running on AWS with these web pages um and it actually worked quite well like you can ask it General common knowledge history politics current events questions um and it would be able to do a fast lookup in the index feed it into the model um and and it would give like a surprisingly good result and so when I saw that I thought that this is definitely doable and like it kind of shocked me that like no one else was doing this and so this was now the fall of 2020 um and um um yeah I was kind of shocked no one was doing this but it cost a lot of money to keep it up I was still in college there were other things going on I got bogged down by classes and so I ended up shelving this for almost a full year actually um and I returned to it in Fall of 2021 when um big science released t0 um when big science released the t0 models that was a massive jump in the reasoning ability of the model and um it was better at reasoning it was better at summarization it was still a glorified summarizer basically was this a precursor to bloom because Bloom's the one that I know I think Bloom ended up actually coming out in 2022 but bloom had other problems where I think for whatever reason the blue models just were never really that good which is so sad cuz I really wanted to use them but I think they didn't train on that much data um I think they used like the original they were trying to replicate gpt3 so they just use those numbers which we now know are like far below chinchilla optimal and even chinchilla optimal which we can like talk about later like what we're currently doing with the F mod goes yeah it goes way beyond that um but they weren't training enough data I'm not sure how that data was clean but it probably wasn't super clean and then they didn't really do any fine tuning until much later um so t0 worked well because they took the uh T5 models which were um closer to chinchilla optimal cuz I think they were trained on also like 300 something billion tokens similar to to gpt3 but the models were much smaller um uh so the models yeah they were pre-trained better and then they were fine-tuned on um this I think t0 is the first model that did large scale instruction tuning um from diverse data sources in the fall of 2021 um this is before instruct GPT um this is before flant 5 which came out in 2022 this is I think the very very first um at least well-known example of that um and so it came out and then I did on top of t0 I also did the Reddit um Eli 5 fine tune um and that was the first model and system that actually worked well enough to where I didn't get discouraged like I did previously because the failure cases of like the BART based system was so egregious like sometimes it would just misinterpret your answers so or questions so horribly um that like it was just extremely discouraging but for the first time like it was working reasonably well um also using a much bigger model I think the BART model is like 800 million parameters but t0 we were using 3B so it was t0 3B you know bigger model um and that was the very first iteration of hello um so ended up doing a show hn on Hacker News in January 2022 of that system our fine tune teas Z Model connected to our elastic search index of those um 350 million top 10,000 common C websites um and to the best of my knowledge I think that's the first um example that I'm aware of of a um llm search engine model that's effectively connected to like a large enough index that I could consider like an internet scale um so so I think we were the we were the first um to to release like an internet skill llm powered rag search system um in January 2022 and around the time uh me and my future co-founder Justin we were like you know we really why not do this full time like this this seems like the future this is really cool um I couldn't really sleep even like I would I was going to bed and I was like I was thinking about it like I would stay up until like 2:30 a.m. like reading papers on my phone in bed go to sleep wake up the next morning at like 8 and just be super excited to keep working um and I was also doing my thesis at the same time uh my senior honors thesis at UT Austin um about something very similar um we were researching factuality um in abstractive question answering systems um so a lot of overlap with this project um and the conclusions of my research actually kind of helped guide the development path of of hello in that in the research we found that llms don't um they don't know what they don't know so the conclusion was is that you always have to do a search um to ensure that the model actually knows what it's talking about um and my favorite example of this even today is kind of with um chat GPT browsing where you can ask chat GPT browsing um how do I run llama.san and you're all good um it won't even bother doing a lookup even though I'm sure somewhere in their internal prompt you know they have something like if you're not sure do a look up like that's not that's not good enough so models don't know what they don't know you always have to do a search um and so we approached llm powered question answering from the search angle um we pivoted to make this for programmers in June of 2022 around the time that um we were getting into YC we realized that like what we're really interested in is the case where the models actually have to think cuz up until then the models were kind of more glorified summarization models like we really thought of them like um the Google featured Snippets but on steroids and so we like we saw a future where the simpler questions would get commoditized um and I still think that's going to happen with like Google sge and like it's nowadays it's really not that hard um um to like answer the more basic kind of like summarization like current events questions with lightweight models and that'll only continue to get cheaper over time and so we kind of started thinking about this trade-off where llm models are going to get both better and cheaper over time um and that's going to force people who run them to make a choice either you can run a model of the same intelligence that you could previously for cheaper or you can run a better model for the same price and so someone like Google once the price kind of Falls low enough they're going to deploy and they're already doing this with that g they're going to deploy um a relatively basic kind of glorified summarizer model that can answer very basic questions about like current events like who won the Super Bowl like you know what's going on on Capitol Hill like those types of things um and the the flip side of that is like more complex questions where like you have to reason and you have to solve problems and like debug code um and we realized like we we're much more interested in kind of um going along the bleeding edge of that Frontier case and so we've optimized everything that we do for that um and and that's a big reason of why we've built fin specifically for programmers as opposed to saying like you know we're kind of a search engine for everyone because um as these models get more capable we're very interested in seeing kind of what the emergent properties are um in terms of reasoning in terms of being able to solve complex multi-stop multi-step problems um and I think that some of those emerging capabilities like we're starting to see but we don't even fully understand so as I think there's always an opportunity for us to become more General if we wanted um but we've uh we've been along this path of like what is what is the best most advanced reasoning engine that's connected to your code base that's connected to the internet that we can just provide what is find today pragmatically from a product perspective how do people interact with it what does it plug into your workflow yeah so find is really a system um find is a system for programmers when they have a question or when they're frustrated or when something's not working they're frustrated yeah for them to get unblocked the most abstract page for find is like if you're experiencing really any kind of issue as a programmer will solve that issue for you in 15 seconds as opposed to 15 minutes or longer and so find has an interface on the web um it has an interface in vs code and more idees to come uh but ultimately it's just a system where a developer can paste in a question or paste in code that's not working um and find will do a search on the Internet or they will find other code in your codebase perhaps that's relevant um find will find the context that it needs to answer your question and then feed it to a reasoning engine powerful enough to actually answer it so that's really the philosophy behind finding it's a system for getting developers the answers that they're looking for um and so right now from a product perspective this means that um we're really all about getting the right context so um the vs code extension that we launched recently is a big part of this uh because you can just ask a question um and it knows where to find the right code context um in your code it can do an internet search as well so it's up to date um and it's not just reliant on what the model knows um and it's able to like figure out what it needs by itself um and answer your question based on that um and if you know it needs some help you can also get yourself kind of just there's opportunities for you yourself to put in all that context in um and but the issue is also like not everyone um wants to use vs code um some people like you know are real neovim sticklers um or you know they're using like pie charm or other IDs um jet brains um and so for those people um they're actually like okay with switching tabs at least for now if it means them getting their answer um because really like there's been an explosion of all these like startups doing code uh doing search Etc um but really who everyone's competing with is chat gbt uh which only has like that one web interface and like chat gbt is really the bar um and so and so that's what we're what we're up against and so your idea you know we have I'm on from cursor on the podcast and they've gone through the we need to own the IDE thing yours is more like in order to get the right answer people are happy to like go somewhere else basically they're happy to get out of their IDE and that was a great podcast by the way uh but but yeah so so part of it is that um people sometimes perhaps aren't even in in an IDE so like Co like the whole task of software engineering goes Way Beyond just writing code right there's also like a design stage there's a planning stage a lot of this happens like on whiteboards it happens in notebooks um and so the web prodct also exists for that where you're not even coding yet and you're just trying to get like a more conceptual understanding of what you're trying to build first um but some I the podcast with them on was great but somewhere where I disagree with them is that um you actually need to own the IDE um I think in the long sorry sorry uh let's cut that yeah so I thought the podcast with the man was great but somewhere where I disagree with them is that you need to own the IDE um I think like he made kind of some good points about you know not having platform risk in the long term but some of the you know features that were mentioned like um suggesting diffs for example um those are all doable with an extension um we haven't yet seen um with vs code in particular um any functionality that we'd like to do yet in the IDE that we can't either do through directly supported vs code functionality or something that we kind of hack into there uh which we've also done a fair bit of um and so I think it remains to be seen um where that goes but I think what we're looking to be is like we're not trying to just be in an IDE or be an IDE like find is a system that goes be on the IDE and like is really meant to cover the entire um life cycle of a developer's thought process in going about like hey like I have this idea and I want to get from that idea to a working product and so then that's what the long-term Vision to find is really about is starting with that or like in the future you know in the future I think programming is just going to be um really just the problem solving like you come up with an idea you come up with like the basic design for that the algorithm in your head and you just tell the AI hey just like just do it just make it work um and that's what we're building towards fantastic um I I I think we might want to give people u in some impression of about like the type of traffic that you have um because when you present it with a text box you could type in anything and I don't know if you have some mental categorization of like what are like the top three use cases that people tend to call L yeah that's a great question um so the two main types of searches that we see are how to questions like how to do X using Y tool um and this historically has been our bread and butter because uh with our embeddings like we're really really good at just going over a bunch of developer documentation and figuring out exactly the part that's relevant and just telling you okay like you can use this method but as lmms have gotten better and as we've really transitioned to um using gp4 a lot in our product um people organically just started pasting it code that's not working and just said fix it fix yeah and what really shocks us is that um a lot of the people who do that um they're coming from chat gbt so they tried it in chbt with chat bt4 it didn't work uh maybe it required like some multi-step reasoning maybe it required um to like some internet context or something found in either a St flow post or some documentation to solve it um and so then they paste it into find and then find Works um so those are really those two different cases like how can I build this conceptually or like remind me of this one detail that I need to to build this thing or just like here's this code fix it um and so that's what a big part of our vs code extension is is like enabling a much smoother here just like fix it for me type of workflow that's that's really its main benefit it's like it's in your codebase it's in the IDE e it knows how to find the relevant context to answer that question um but at the end of the day like I said previously that's still a relatively not to say it's a small part but it's a limited part of the entire kind of mental life cycle of of a programmer Y when you launched in so you launched in Feb and then you launched V2 in August you had a couple other pretty impactful post SL feature launches um the web search one was was massive yeah um and you you are so you are mostly a GPT 4 rapper we were for a long time for a long time until recently yeah until recently um so like people coming over from chbt were saying ask according to same model yep uh what with your version of web search would that be the primary value proposition basically yeah and so what we've seen is that any model plus web search is just significantly better than that model itself you think that's what you got right in April like um so you got, 1500 points on Hacker News on April which is UN like if you live on Hacker News a lot that is unheard of for someone so so early on in your in your journey yeah we super super grateful for that definitely was not expecting it so what we've done with Hacker News is we've just kept launching yeah like uh what they don't tell you is like you can just keep launching so so that's what we we've been doing so we launched the very first version of find um in its current Incarnation um after like the previous demo connected to our own index like once we got into IC we scrapped our own index cuz it was it was too cumbersome at the time um we moved over to using Bing as kind of just the raw Source data and uh we launched as hello cognition and over time every time we like added some intelligence to the product to a better model we just keep keep launching and every additional time we launched we got way more traffic so we actually silently re rebranded to find yeah in late December of last year but like we didn't have that much traffic nobody really knew who we were how' you pick the name by Paul Graham actually picked it for us all right tell the story yeah so oh boy yeah where do I start so this is the biggest side should I should we go for like the full the full program story or just you want to do it now or you want to do it later I'll give you a choice I think okay let's let's just start with the name for now and then we can do the full Paul gr story later um but basically um Paul Graham when we were lucky enough to meet him he saw our name and our our main was at the time say hello doso and he's just like guys like come on like like like like what like what is this you know like and um and we were like yeah but like when we bought it you know we just kind of broke college students like we didn't have that much money and like we really liked hello as a name because um it was the first like conversational search engine and that's kind of that's the angle that we were approaching it from and so we had say hello so and he's like there's so many problems with that like like like the say hello like what does that even mean and like doso like it's got to be like aom we did some time just like with Paul Graham in the room we just like looked at different domain names like different things that like popped into our head um and one of the things that popped into like Paul Graham said was F like with the pH D spelling in particular yeah which is not typical naming advice right because it's not when when people hear it they don't spell it that way exactly it's it's hard to spell and also it's like very 99s and so at first like we didn't like it I was like like yeah like I don't know but over time like it kind of it kept it kept growing on us and um and eventually we're like okay you know we like the name um it's owned by this elderly Canadian gentleman uh who who got to know and he was willing to sell it to us and so we bought it and uh and we and we changed the name yeah um but anyways where were we we were I had to ask I mean you know everyone who looks at you looks is wondering a lot of people and a lot of people actually pronounce it find um which you know and by now is kind of you know it's it's it's part of the game but eventually we want to buy f.com and then just have that redirect to PhD yeah so PhD is like definitely the right spelling like we'll just yeah we'll have all the cases addressed so being web search uh and then and then in August you launch V2 Could you um is is is V2 the the find as a system pitch or have you Mo evolved since then yes so I don't I like the V2 moniker like I don't really think of it that way in my mind there's like there's the version we launched during last summer during YC which was um the Bing version directed towards programmers um and that's kind of like that's why I call it like the first incarnation of what we currently are because it was already directed towards programmers we had like a code snippet search built in as well because at the time you know the models we were using weren't good enough to generate Cod Snippets even GPT like the Texas Da Vinci 2 which aail was available at the time wasn't that good at generating code and it would generate like very very short very incomplete um code Snippets and so um we launched that last summer got some traction but really like we were only doing like I don't know maybe like 10,000 searches a day like some people knew about it some people used it which is impressive because looking back the product like was not that good um and yeah every time we've like made an improvement to uh the way that we retrieve context uh through better embeddings more intelligent like HTML parsers um and importantly like better underlying models um yeah I would really consider every kind of iteration after that when we every major version after that was when we introduced a better underlying answering model like in February we launched um we it took it took a it's we had to swallow a bit of our pride when we were like okay our own models aren't good enough we have to go to open AI um um and that actually that did lead to kind of like our first like decent bump of traffic um in February um and people kept using it like our attention was way better too um but we were still kind of running into problems of like more advanced reasoning some people tried it but people were leaving because even like GPT 3.5 uh both turbo and non-turbo like still not that great at doing like code relas reasoning Beyond uh like the how do you do X like documentation search type of use case um and so it really only when GPT 4 came around in April that we were like okay like this is like the our first real opportunity to really make this thing like the way that it should have been all along um and having gp4 as the the brain um is what led to that Hacker News Post um and so what we did was we just let anyone use gp4 on fine for free without a login um which I actually don't regret so it was very expensive obviously but like at that stage all we needed to do was show like we just needed to like show people here's what fine can do that was the main thing and so that worked that worked like we got a lot of users um um do you know fireship yeah YouTube channel YouTu Jeff Delany yeah he made a uh a short about find oh and that's and on top of The Hacker News Post and that's what like really really made it blow up it got millions of views in days and and he he's just funny like what I love about fire ship is like he like you guys yeah you like humor humor goes a long a long way um towards like really grabbing people's attention and so that blew up so some something I would be anxious about as a Founder during that period so obviously we all remember that pretty closely there were a couple of people who had access to the GT4 API doing this which is unrestricted access to GT4 and I have to imagine YC uh opening I wasn't that happy about that uh because it was like kind of de facto access to GT4 before they released it chat4 was in chat PT from day one I think um open AI actually came to our support because what happened was we had people building unofficial apis around to try to get free access to it um and I think opena actually has the right perspective on this where they're like okay people can do whatever they want with the API if they're paying for it like they can do whatever they want but it's like not okay if you know paying customers are being exploited by these other actors so they actually got in touch with us and they helped us like um set up better cloudflare bot monitoring controls um to effectively like crack down on those unofficial apis um which yeah which you know we're very happy about um but but yeah so so we launched gp4 a lot of people come to the product um and yeah for a long time we're just we're figuring out like how do we like what do we make of this right like how do we a make it better but also deal with like our costs which have just like massively massively ballooned and I think um over time it's I think it's become more clear with the release of llama 2 and llama 3 on the horizon that we will once again see a return to um vertical applications running their own models as was true last year and and before um I think that gp4 my hypothesis is that the jump from 4 to 4.5 or four to five will be smaller than the jump from 3 to Four 3 to four and the reason why is because there were a lot of different things like there there was two plus effectively two two and a half years of research that went into going from 3 to four like more data bigger model all of like the instruction tuning techniques rhf um all of that is known and like meta for example and now there's all these other startups like mrr too like there's a bunch of very well-funded open source players that are now working on just like taking the recipe that's now known and scaling it up so I think that even if Delta exists in 2024 the Delta between proprietary and open source won't be large enough that a um startup like us with a a lot of data that we've collected can take the data that we have f- tune an open source model and like be able to have it be better than whatever the proprietary model is at the time that's that's my hypothesis that will once again see a return to these verticalized models um and and that's something that we're super excited about cuz um yeah that brings us to kind of the fine model because um that the plan from kind of the start was to be able to return to that if that makes sense and I think now we're definitely at a point where it does make sense um because we have requests from users who like they want longer context in the model basically like they want to be able to um ask questions about their entire code base um they want and without you know context and retrieval and taking a chance of that like I think it's generally been shown that if you have the space to just put the raw files inside of a big context window that is still better than chunking and retrieval it just it just is so there's various things that we could do with longer contacts faster speed lower cost uh super excited about that and that's the direction that we're going with to fine model um and our big hypo hypothesis there is is precisely that we we can take an really good open source model um and then just train it on absolutely all of the high quality data that we can find um and there's a lot of various you know interesting ideas for this we have our own techniques that we're kind of playing with internally one of the very interesting ideas that that I've seen is octop Pac from um from Big code I don't think that it made that big waves when it came out I think in August but the idea is that um they have this data set that Maps um GitHub commits um to a change so basically there's all this really high quality like human-made human human written diff data out there on every time someone makes a commit in some repo um and you can use that to train models uh you take the file state before and like give in a commit message what should that code look like in the future do you think your money is any good no unfortunately so so we ran this experiment we trained the fine model um and if you go to the big code leaderboard um as of today October um 5th um all of our models are at the top of the um big code leaderboard by far it's not close particularly in languages other than python um we have a 10o gap between us and the next best model uh on Java JavaScript I think c um multilingual um and what we kind of learned from from that whole experience um releasing those models is that human eval doesn't really matter um not just that but gp4 itself has been trained on human eval and we know this because gp4 is able to predict the exact dock string in many of the problems um I've seen it predict like the specific example values in the do string which is extremely improbable for it to just you know know um so I think there's a lot of data set contain contamination and it only captures a very limited subset um or like what programmers are actually doing um what we do internally for evaluations are um we have gp4 um score answers uh gp4 is a really good evaluator I mean obviously it's by really good I mean it's the best that we have I'm sure that you know a couple months from now next year we'll be like oh you know like gbt 4.5 gbt 5 it's so much better like gb4 is terrible but like right now it's the best that we have short of humans um and what we found is that when doing like temperature zero um evals um it's actually mostly deterministic gb4 um across runs um in assigning scores to to different answers so we found it to be a very useful tool in comparing our model to say gp4 um but yeah on our like internal like real world here's what people will be asking this model data set um and the other thing that we're running running is just like releasing the model to our users and just seeing what what they think uh cuz that's like the only thing that really matters is like releasing it for the application that it's intended for and then seeing how people react um and for the most part the incredible thing is is that people don't notice a difference between our model and gp4 for the vast majority of of searches there's some reasoning problems um that gb4 can still do better we're working on addressing that um but in terms of like the types of questions that people are asking on find um yeah like there's there's not that much difference and in fact like i' I've been running my own kind of side by-side comparisons um shout out to God mode by the way and I've like myself have kind of confirmed this to be the case and even sometimes it gives a better answer um perhaps like more concise or just like better implementation than gp4 which that's what surprises me um and and so we like by now we kind of have like this reasoning is all you need kind of hypothesis where we've seen emerging capabilities in the fine model where by training it on high quality code it can actually like reason better um it went from not being able to solve um like world problems where like riddles were like with like temporal um and like like placement of objects and moving and stuff like that uh that gp4 can do pretty well we went from not being able to do those at all to being able to do them just by training on more code which is wild um so so we're already like starting to see like these emerging capabilities yeah so I just wanted to make sure that we have the I guess like the the the model card in our heads so you started from Cod Lama yes uh 65 34 34 so unfortunately there's no Cod Lama 70b if there was that would be super cool but there's not 34 and then uh which which by which in itself was Lama 2 uh which is on two Chillin tokens and he added 500 billion code tokens yes and just added a bunch more yeah and they all they did also did a couple of things so they did I think they did 500 billion like General pre-training and then they did an extra 20 billion long context pre-training so they actually um ra increased the like Max position um tokens to 16k up from 8K um and then they changed the um the Theta parameter for the ROP embeddings as well um to give it theoretically better long contact support up to 100K tokens uh but yeah but otherwise it's like basically l so you just took took that and just added data exactly you didn't do any of other fundamental yeah so so we didn't actually we we haven't yet done anything with the model architecture and we just trained it on like many many more billions of tokens y um on our own infrastructure um and something else that we're taking a look at now is um using reinforcement learning for correctness um one of the interesting pitfalls that we've noticed with the fine model is that in cases where it get stuff wrong sometimes is capable of getting the right answer it's just there's a big variance problem it's wildly inconsistent um like there are cases when it is able to get the right Chain of Thought and able to arrive at the right answer um but not always and so like one of our hypothesis and something that we're going to try is that like we can we can actually do reinforcement learning on like for a given problem generate a bunch of completions and then like use the correct answer as like a loss basically to try to get it to um um be more correct and I think there's a high chance I think of this working because it's very similar to the like rhf method where you basically show um pairs of completions for a given question um except the criteria is like which one is like you know um less harmful um but here you know we have a different criteria but it if the if the model's already capable of getting the the right answer which is which it is we're just we just need to cajo it into being more consistent there were a couple things that I noticed in the product that were not strange but unique so first of all the model can talk multiple times in a row like most other applications is like human model human model and then you had outside of the thumbs up thumbs down you have things like have D llm prioritize this message in its answers or then continue from this message to like go back how does that change the flow of the user and like in terms of like prompting it um yeah what are like some tricks or learnings you had yeah that's a great question um so so yeah that's specifically in our pair programmer uh mode which is a more conversational mode um that um also like asks you clarifying questions back if it doesn't fully understand what you're doing and it kind of it it holds your hand a bit more um and so from user feedback we we had requests to make make more of an auto GPT where you can kind of give it this problem that might take multiple searches or multiple different steps like multiple reasoning steps to solve um and so that's the um that's the um impetus behind building that product uh being able to do multiple steps and also be able to handle really long conversations like people are really trying to use the pair programmer to go from like sometimes really from like basic idea to like complete working code and so what we noticed was is that we were we were having like these very very long threads with like 60 messages like 100 messages and like those become really really challenging to manage like the appropriate context window of what should go inside of the um inside of the context um and how to preserve the context so that the model can continue or the product can continue giving good responses um even if you're like 60 messages deep in a conversation um so that's where the the prioritized user message or like comes from is like uh we people have assess to just like let them pin messages that they they want to be left in the conversation um and and yeah and and then that seems to have like really gone a long way towards solving that problem yeah and then you have a run and ret thing are you planning to build your own rapple like learning some people trying to run the wrong code unsafe code yes yes so I think like in in in the long-term vision of like being a place where people can go from like idea to like fully working code having a code sandbox like a natively integrated code sandbox makes a lot of sense um and repet is great and people use that feature um but but yeah I think there's more we can do in terms of like having something um a bit closer to code interpreter where it's able to run the code and then like recursively iterate on it exactly I think replit is working on um apis to enable you to do that yep so AMJ has specifically told me in person that he's he wants to enable that for people at the same time he's also working on his own models right and Ghost Rider and you know all the other stuff yeah so it's kind to get interesting like he wants to power you but also compete with you yeah and like and we love repet um I think that a lot of these like a lot of the companies in our space like we're all going to converge to solving a very similar problem but from a different angle so like repet um approaches this problem from the IDE side like they started as like this IDE that you can run the browser um and they started like from that side making coding just like more accessible and we're approaching it from the side of um like an llm that's just like connected to everything that it needs to be connected to which includes your code context so that's why like we're kind of making you know inroads into idees um but we're kind of we're approaching this problem from different sides and I think it'll be interesting to see where things end up um but I think that you know in the long long term we have an opportunity to also um just have like this General kind of like technical reasoning engine product um that's you know potentially also not just for not just for programmers it's also powered in this web interface like we're there's there's potential I think other um things that we will build that eventually might go beyond like our current scope exciting we'll look forward to that thank you uh we're going to zoom out a little bit into sort of AI ecosystem story but but first we got to get the Paul gri Ron Conway story yeah so um flashback to last summer we're in the YC batch um and uh we're doing the summer summer batch summer 22 so the summer batch runs from June to September approximately and so this was late July early August um right around the time that many like YC startups start like going out like geing up how here's how you know we're going to pitch investors and everything um um and at the same time me and my co-founder Justin we were planning on moving to New York um so um for a long time actually we were thinking about building this company in New York um mainly for personal reasons actually because like during the pandemic pre chat gbt pre- last year pre the AI boom um SF unfortunately really kind of you know like so did lost its luster yeah like no one was here um it was far from Clear like if there would be an AI boom if like a SF would be like the AI yeah exactly if SF would be so back as everyone is saying these days um it was far from clear and so and all of our friends from we were graduating college um cuz like we happened to just graduate college and immediately start YC like we didn't even have I think we had a week in between um so it's you didn't bother looking for jobs you were just like this is what we um well actually both me and my co-founder we had jobs that we secured in 2021 from previous censorships but we both like we funny enough um I when I spoke to uh my boss's boss at the company at which like at the where I reneged my offer I told them we got into YC um they actually said yeah you should do YC wow that's very selfless that's great yeah that that was really great that they did that in San Francisco they would have offered to invest as well yes yes they would have uh but yeah but we were both planning to be in New York um and all of our friends were there from college and uh so like at this point like we have this whole plan we're like on August 1st we're going to move to New York and we had like this Airbnb for t