No Priors Ep. 57 | With LangChain CEO and Co-Founder Harrison Chase

Key Takeaways

hi listeners and welcome to another episode of no priors today we're talking to Harrison chase the CEO and co-founder of linkchain a popular open source framework and developer toolkit that helps people build llm applications we're excited to talk to Harrison about the state of AI application development the open source ecosystem and its open questions welcome Harrison thanks for having me I'm excited to be here Lang Chain's a a really unique story and it started actually as a personal project for you can you talk a little bit about what what Lang chain is and what it was originally yeah absolutely so how how would answer the question what Lang chain is has kind of evolved over time as as the entire landscape Lang chain the open source uh package started yeah as a side project um so so my backgrounds in ml and mlops I was at I was at my previous company I I knew I was going to leave I didn't know what I was going to do this was in September October of 2022 um and so went to a bunch of hackathons went bunch of meetups chatted with folks that were playing around with llms um and saw some Comet abstractions put it in a python project as a just fun side project turned out to strike a chord be fantastic timing you know chat GPD came out like a month later um and it's kind of evolved from there so right now linkchain the company um there's really two main products that we have one is the Lang chain open source packages and happy to dive into that more and then the other is Lang Smith a platform for for testing evaluation monitoring and and all of those types of things and so you know what Lang chain is has evolved over time as yeah the company's grown one thing that we talked about the last time we saw each other in person was just how quickly like the AI um ecosystem and research field is evolving and what it means to manage an open source project through that can you talk a little bit about what you decide to keep stable and change when you both have like big ecosystem of users now and like very rapidly changing environment of applications and Technology that's been a fun exercise so I mean if we go back to the original version of Lang chain what it was when it came out was essentially three kind of like highlevel implementations two were based on research papers and then one was based on natat fredman's like gbot type of agent web crawler thing and so there was some high level kind of like abstractions and then there was a few like Integrations so we had Integrations with I think like open Ai cohere and hugging face to start or something like that and those two layers have kind of like remained so we have you know 700 different Integrations we have a bunch of kind of like higher level chains and agents for for doing particular things I think the thing that we've put a lot of emphasis in um to your point around kind of like what's remained constant and what's remains uh and what's changed is like a lower level kind of like abstraction and runtime for for joining these things together one of the things that we pretty quickly saw was that as people wanted to improve the performance go from prototype to production they wanted to customize a lot of these bits and so we've invested a lot in uh a lower level kind of like chaining protocol so laying chain expression language and then in in a different protocol laying graph which is one something we're really excited about and that's more aimed at uh basically graphs that are not dags so you know all these agents are basically running an llm in a loop you need Cycles um and so Lane graph helps with that and so I think what we've kind of seen is underlying bits of um there's all these different Integrations and like you know there's there's LMS Vector stores and sometimes they change right when chat models came out like that was a that's a very big change in the API interface and so we had to add a new abstraction for that but those have especially over the past few months remained relatively stable um we've invested a lot in this underlying runtime which emphasizes a few things uh streaming structured outputs and and the importance of those has remained relatively stable but then the way that you put things together and the kind of like patterns um for building things has definitely evolved over time from like simple chains to complex chains to then these kind of like autonomous agents to now something um maybe in the middle of like complex State machines or graphs or something and so it's really that upper layer which is like the common ways to put things together that I think we've seen the most rapid kind of like churn what do you think is still missing from uh really getting to performing agents there's a number of companies that have been started recently that are really focused on sort of the agentic world and pushing that whole thread in certain types of automation forward what do you use the big components that you all don't have or that maybe the industry more generally doesn't have that that still needs to come into place to help drive those things ahead yeah that's a that's a really good question I think there's a few things one I think like um like figuring out the right ux for a lot of these things is still an open question in my mind um and you know that's not necessarily something we can help with I think there's a lot of exploration that applications need to do to figure out how to you know communicate what these agents are good at and bad at to end users and expose ways to um maybe let them course correct and see what's going on and so you know I think we try to emphasize a lot of this um observability of intermediate steps and even correcting intermediate steps but but there's a lot of experimentation around ux that I think needs to happen um another big part I think is is basically the planning ability of the underlying llms um I think that's probably the biggest I think when we see people building agents that work right now it's often breaking it down into a bunch of smaller components and and kind of like imparting their domain knowledge about how information should flow through these components um because I think the Els by themselves still aren't able to to reason fully about how that should happen and I think we see a few kind of like uh a lot of research is actually around this I would say in the academic space specifically I think there are two different types of research papers around agents that we see we see some around like planning for agents so there's a bunch of papers that do kind of like an explicit planning Step Up Front um and then there are uh other research papers that do a bunch around reflection so like after it after uh an agent does something is this actually right how can I kind of like um you know improve upon that and I think both of those are basically trying to get around the shortcomings of llms and that in theory they should do that automatically right like you shouldn't have to ask an llm to plan or to think about whether what it's done is correct it should know to do that and then it can kind of like run in a cycle but we see a lot of shortcomings there um and so I think planning ability of llms is is is a big one and that'll get better over time the last one is maybe a little bit more vague but I think even just as Builders we're still figuring out the right ways how to make all these things work what's the right information flow between all the different nodes um in order to get those nodes which are typically an llm call to work do you want to do F shop prompting do you want to fine-tune models do you want to just work on improving the instructions and the prompt um and so I think there's a lot of uh how how do you test those nodes uh that's a big thing as well how do you get confidence in your llm systems and llm agents um and so I think there's a lot of workflow around that to to kind of like be discovered and figured out one thing that's sort of come up repeatedly Rel relative to agents has just been like memory and so I wasn't sure how you think about memory and implementing that and what that should look like and because it seems like there's a few different Notions that people have been putting forward and I think it's super interesting so I was just curious about your thinking on that I also think it's super interesting um I have a few thoughts here um so I think there's maybe two types of memory and they're and they're related but I'll draw some distinction between kind of like system level procedural memory and then like personalization type memory um so system level memory I mean more like what's the right way to use a tool what's the right way to accomplish this objective independent of of who exactly the person is and how I'm different than Sarah or something like that um and then for the personalization bit I think it's like okay you know Harrison likes soccer and he likes basketball and I should remember that when he asked questions um and so I think there's there's uh maybe slightly different ways that we see teams thinking about both of these so on the procedural side I think the main thing that we see people doing um and that we think is pretty effective is uh few shot prompting and maybe find tuning for how to use uh for how to use tools because that's basically what it comes down to what's the right way to use tools what's the right way to plan and we see F shot examples being really really impactful for that and so that's something where and so there I think there's this really interesting data flywheel of like monitoring your application Gathering good examples um and then and then plugging those back into your application in the form of few shot examples that we're pushing really heavily with laying Smith right now and then the other side of it is just like personalization level memory um and I think there's a few different ways to do this like I think open AI implemented it in their chat uh in their chat uh GPT where it in the way I think it does it under the hood is it basically has functions that it can call to say like remember this fact or delete this fact um and so that's a really interesting like active um active Loop that the agent is engaging in where it explicitly decides what it wants to remember and what it doesn't want to remember I also think one thing that I'm bullish on is a more kind of like uh passive background uh process that kind of looks at conversations and almost like extracts insights um and then you can use those insights in kind of like future conversations and I think there's pros and cons to each and I think it speaks to the memory in general I feel is like a field that's just like super super nent like I don't I actually am am underwhelmed at the amount of like really interesting stuff that's going on there um and so I think you know bunch of different approaches no no kind of like overwhelmingly best solution has the um sophistication shape type of application that you see people building with Lang chain or just generally in the ecosystem dramatically changed over the last few months I do think that there are more examples kind of as elad mentioned of um agentic applications that are much more productive and more sophisticated like multi-step rag systems with much more useful ranking like does that match with the patterns you're seeing or like what are you what are you seeing that excites you the most that you think is most useful that does generally match I think Lang chain from the beginning has always been focused on those types of applications um and and uh not only the open source but also Lang Smith the platform so I think you know a lot of the emphasis that we put into like the testing and the observability is really focused on these like multi-step things we've always been focused on those probably it's generally true in the market that there's been more uh uh of a trend towards those but from our perspective we've always been focused on those and so I think you know that hasn't been as dramatic I think there has been like interesting um things within that that have emerged just calling out like a few things um within rag I think we've seen really interesting and advanced query analysis start to come into play so uh you know you're not just passing the user question directly to an embedding model you're maybe doing some analysis on it to figure out which which retriever should I send it to or like what is the bit that I should search is there kind of like a explicit metadata filter so some and then so that retrieval is like a multi-step uh process and more there um and explicitly around query analysis um f shop prompting and that whole data flywheel I think we're starting to see come into play more on the agent side um I kind of alluded to this earlier but I think you know um the way that we've kind of thought about things is there's kind of like chains which are sequential steps you're going to do this and then you're going to do this and then you're going to do this and you're always going to do those in the exact sequence and then you know last March or April or whenever Auto GPD came out and it was like we're literally just going to run this in a for Loop and it's going to be you know this autonomous agent and I think the things that we see making it into production and and informed um a lot of the development of Lange graph um is are is something in the Middle where it's like this controlled State machine type thing um and so we've seen a lot of that come out recently and so i' maybe call out that as like one um thing that we've really updated a lot of our beliefs on over the past few months yeah I think a combination of that and Tre search and just like trying to be efficient with like your sampling at every step has shown like a lot of really interesting uh effective applications recently and I think the like cognition as one example of like a surprisingly amazing agent has has come out like where else do you think agent agentic applications will begin to work or that you've already seen I think on the customer support side that's a pretty obvious use case I think Sierra um you know has emerged there and is doing is doing quite well there um I think yeah the cognition demo was very impressive I think they did a lot of things right I think they really nailed a really interesting ux um and that was maybe one of the things that that I was most excited about um and then obviously it seems to work very well and so I I don't know exactly what they're doing under the hood um uh but but those type like coding coding problems in general we see a lot of people working on I think you there's a really nice feedback loop that you can get by just like executing the code and seeing if it works um and you know as well as the fact that people building it are developers and so they can they can uh test it um coding customer support there there's some interesting stuff around like recommend like recommendation um chat Bots almost um so I draw a distinction between that and customer support or with customer support you're maybe trying to explicitly kind of like resolve a ticket or something like that and the um and the recommendation bit is a bit more focused on like a user's preferences and and what they like um and I think we've seen a few uh I think we've seen a few things emerge there um but I'd say customer support and coding are the two Clara as well you know they came out and and had a pretty good release one um pattern that I think is very popular and I can't tell if it is real or transient is whether or not companies will be able to switch between different um llm models right whether it's a you know self-hosted like dedicated um uh inference um you know instance for for them or if it's an actual API provider but for any given application take your prompts and go from you know um anthropic to mraw to open AI to something else um in reality it feels like you know the way an application uh responds is probably going to be sensitive to the fact that these LMS are actually going to predict differently like what do you think about this can you can you switch is that a real pattern it's not as easy as it seems like it should be and I think the main main thing is that the prompts still need to be different um for each model I do think um the prompts will probably start to converge in the sense that if you think the models are getting more and more intelligent then like hopefully these small idiosyncratic EES don't matter as much um and as more and more model providers start supporting the same things um then that will make it easier and what I mean by that is you know so many prompts for open AI which is you know the leading and most used one use function calling um and you know up until some period ago like no other models did and so you just like couldn't use those prompts at all um but now like mraw has function calling and and and Google has function calling and so I think they're a little bit more transferable there what else is on that list there's function calling there's visual input like what else is going to differentiate these um model apis context Windows one as well so I think this gets to like yeah what's the right context that you can be passing if it's longer you know if that changes then changes that doesn't like that changes the whole architecture of your application um modalities one prompt injection for safety yeah I I think that's interesting um I think that's a real Enterprise concern I think a lot of the agents are still just figuring out how to make agents work this is a different axis almost but to the point around like switching models I do think we see a desire for this especially when you start going to scale um so I think it's like make something work with gp4 but then okay you're rolling it out are you really is that like you know are you really going to eat that much cost with gp4 can you use GPD 3.5 do you want to fine tune and so I think that's that like that transition is where we really start to see people um thinking about switching models um there's definitely some switching models at the beginning like if you just want to play around with different models and see their capabilities but I think the most like pressy need to switch models happens when you go from prototype to to scale cost and latency would be differentiators there as well one thing you mentioned I thought was really interesting is just context windows and obviously Gemini launched with um a million token context window and I was just curious um how you think about context window versus rag versus other aspects of the model and how all those things tie together and you know once we get to very long context windows and the tens of millions of tokens like does that really shift things radically or how does that change functionality and so I was just curious since you've thought about how all these things piece together um I was just curious how you think about those different factors and what they mean very good question that a lot of people are thinking about who are a lot smarter than me I think um I mean a few thoughts I think like longer context Windows definitely make like single shot things much more realistic um like extraction of elements in a long PDF you can do that one shot um rag over a single long PDF or like five long PDFs okay cool you can do that you can do that one shot there I think um there are definitely things at scale that don't fit um you know into a single uh uh context window there are also things where it requires iterations you need to like decide what to do interact with the environment get that back so this whole idea of chaining um and agents I don't like that's that's less around context windows and more around interacting with the environment and getting feedback and and so I don't think that's going anywhere um I think with respects to rag in particular because I think that's where it often comes up like you know did this kill rag um I think there's a few things actually just today one of our team members Lance Martin there's that like everyone's doing the needle in the Hast stack thing and now all these models are like green across the board for whatever reason they've all figured it out um but I think like that that actually really doesn't reflect a lot of rag use cases in in my opinion because like that's the needle in the Hast stack is like okay given this long context can I find a single information point but often times rag is about seeing multiple information points and then reasoning over them and so I think with the Benchmark he released is exactly that like as you increase the number of of needles um you know performance goes down as you might expect and then also when you ask it to reason rather than just retrieve the performance drops as well and so there I think there's more work to be done there and then I think another thing is just around the ingestion for rag in the in the indexing process like a lot of attention has been paid to like um Tex splitting and chunking and and and all of that and I don't know exactly how that will change like will you still do that but you now just retrieve the whole document like we have a concept in Lang chain of like a parent document retriever which basically creates multiple vectors for for each document so maybe you just do that maybe you still maybe you chunk it up into larger chunks and just retrieve those larger trunks maybe use a a traditional search engine like elastic search or something I'm not I'm I'm not sure that's probably the place I have the least confidence in the one other area that I see a lot of people talking about and I see a fewer people actually doing uh is fine tuning and um to some extent I think that's because with fine tunes you lose generalizability and so people just start focusing on prompt engineering or other ways to effectively get the same performance without the actual fine tune but it's something that feels very um Aaron and people talk about it a lot and people talk about doing it a lot um You probably have a great perspective since you see so many different types of customers are are you seeing a lot of fine tuning happening in the wild and if so there's specific common applications or use cases for it we see people experimenting with it I think the only real place where they're doing it is when they've reached like really critical scale um which I still don't think is that many applications to to date um I think there's a lot of difficulties with it um one's like Gathering the data set for it and so I think a lot of the things we have in link Smith tackle a lot of these issues but like Gathering the data set for it um so like having that data visibility and starting to curate that data set um evaluating the fine-tuned model um so like evaluation and testing is is a huge pain point there that we're trying to tackle in a few ways the third is just like yeah back to this point of people are still just like experimenting so rapidly it's much harder to change a fine-tuned model than it is to change a prompt or even changed few shot examples and so I think we're seeing more and more people use few shot examples um but not a ton graduating to the fine-tuning just because yeah I think uh much harder to just like iterate quickly on in terms of other major changes in the landscape it's been it's been a big year the first commit to Lang chain I think was in octob October of 22 which is like when I launched um conviction as a fund as well uh at that time we didn't have LL 2 we didn't have mraw there were not um nearly as many open-source models with um what people would consider to be more useful reasoning ability has that changed in terms of like what you see um application developers do with linkchain Gemini too oh and Gemini yeah fun story about that the original models that we launched with open AI actually deprecated like a month ago so the like actual original L chain you can't run because the models don't exist anymore um but yeah um like there's I think we see increasingly interest in open source but the reasoning abilities are still just like lagging behind clae 3 or gp4 um and I think like for a lot of the applications that it kind it probably depends on the types of applications that you're building but a lot of the applications that Lang train is focused on with this kind of like reasoning aspect those are just so crucial um and I don't think we see super compelling um I don't I still don't think we see super compelling reasoning abilities in the open source models and maybe that's one of my hot takes but I think for a lot of the Lang train apps the open source models maybe don't live up to a lot of kind of like the the Twitter hype or Twitter excitement at least not yet zooming out like you have really Broad View like what do you feel like that no one is working on that it's going to enable better applications that should be I think the most exciting stuff is at the application in ux Lay right now I think that's where the most exciting stuff is there one of the uh I don't know if this is this this is maybe this is in more the capabilities side is but like memory I think is super interesting especially like personalized long-term memory um I don't know if I don't know if it's necessarily tooling so much that needs to be built there as it's just like an application in a ux that's really focused on that um and and you know if I if I wasn't doing link chain if I was starting a company right now I'd probably start something at the application layer and it would probably be something that really takes advantage of like long-term memory I guess at the the high level similarly is there anything that you view as like a major prediction or things that'll change over the next year that nobody's really paying as much attention to memory is a big interest of of of ours um and so I hope that will have some kind of like breakthroughs there I think a lot of the specifically around yeah learning from interactions incorporating that back in at a user level um in a similar vein also this uh type of more like system level memory I think is really interesting and building up building towards this idea of almost like continual learning so there's you know like can you learn from your interactions and you can do that in a variety of different ways this may just be where we sit in the ecosystem but one exciting um and probably under talked about ways it's just idea of of building up F shot example data sets and really using those I think it's much faster and cheaper than fine-tuning models um it's easier to do than trying to like programmatically change the prompt in some way like that's still kind of like a a a bit of a art um and so yeah continual towards continual learning with few shot examples is is maybe one like really interesting area that that we're excited about can you help our listeners ex like just imagine like a little bit more vly like what um type of application experience that would enable like you know a consumer application or a business application of what that type of continuous learning would allow you to do yeah absolutely I think at a high level it would basically allow the application to automatically get better over time and it could get better in the sense that it's just more accurate so you know it's it maybe you know it first does a mistake you then like tell it that it made a mistake and it automatically kind of like incorporates that either as a few shot example or update to a prompt but it starts learning from its its mistakes and its successes as well right there's a really cool project called dpy or DSP I don't know how to pronounce it um but it's out of Stanford I say dispy oh no there's there's three ways now I say the aspa no I'm just kidding so and I think that actually tackles like I actually see a lot of similarities between that and Lang chain Lang Smith in some way and I think it's all towards this idea of like so so so dpy or disp or or whatever um is basically this idea of like optimization you have kind of like inputs outputs you then have your application which uh they similarly think as as like multiple steps and you basically uh you basically optimize your your application through a variety of different ways the main one of which I would say is probably F shot examples so they we'll probably do a webinar with Omar and he can correct me if I'm wrong um and I think the idea of like continual learning is basically doing that optimization but in an online manner where your feed you don't have like ground truth necessarily but you get feedback from the environment thumbs up thumbs down if if things are good and so I think yeah that kind of like optimization Loop whether offline or online is really really exciting and I think a similar thing could maybe I think you can think of like personalization also as like what this would look like to end users and and maybe like consumer facing apps so you start with like a generic application that does the same thing for everyone but then it maybe learns to to search the web differently for for me and elad or something like that um and so I think that's like concretely how it could could manifest cool thanks so much for doing this um it's obviously a pleasure to have you on no thank you guys good to see you find us on Twitter at no prior pod subscribe to our YouTube channel if you want to see our faces follow the show on Apple podcast Spotify or wherever you listen that way you get a new episode every week and sign up for emails or find transcripts for every episode at no- prior .c