OpenGPTs

Key Takeaways

hello today I want to talk about open gpts open gpts is a project that we built here at linkchain uh that replicates the GPT store in a few ways so it creates uh end user-facing friendly interface to create different Bots and these Bots can have access to different tools and they can uh be given files to retrieve things over and basically it's a way to create a variety of bots and expose the configuration of these Bots to end users it's all open source um it can be used with open AI it can be used with other models as as we'll see um and it's an exciting way to create a a GPT store like experience if you're building a more focused platform an internal platform or any of that so we launched this a few months ago actually right when uh open AI released their GPT store and but we haven't really dove into what's going on or how to use it um and so there's several things that I want to cover in this video there's maybe two main areas one I want to talk about it as an enduser facing application so how can you interact with we we have a we have a simple research preview hosted version of this um how can you interact what can you do what the functionality in the second half of the video I want to talk about how openg gpts is built um we do not intend to uh you know monetize openg gpts we want this to be a platform that people can clone and spin up their own versions of the GPD store or an internal platform or things like that so the second half of this video will focus more on how to build this platform and some of the considerations that we made please note that I am filming this on January 30th we are going to continue working on this continue changing it so if you are watching it later the hosted version or the code may have changed so starting with how to use it um the the the first thing that you'll want to do is create a bot and we actually have three different types of bots that one can create I'm going to start with assistant which is the default bot and and is also the most powerful one so the assistant can use an arbitrary number of tools and you can give it arbitrary instructions the llm will then take those instructions and those tools and decide which tools if any to call um get back the response and then and then continue on its way so let's create one that has access to the internet um so let's name it internet bot um let's give it some instructions you are a helpful weatherman always tell a joke when giving the weather let's scroll down there's a bunch of tools that we can use I'm going to use the search by tavil um they're a LM focused search engine um and then the currently the only agent type that we have supported in the research preview is with GPT 3.5 turbo but we will show in the code when you're building this you can enable CLA you can enable uh Google Gemini you can use gp4 you can use Azure you can use Bedrock but this is just the one that we have in the research preview so I'm going to save uh this now I'm going to try chatting with it let's say hi so several things to notice um one it's streaming responses so we put a lot of work into streaming of tokens this is important for uh uh you know most chat-based applications um it's got some feedback as well so I can give it thumbs up thumbs down um and start recording feedback and and when we talk about the platform side of things we'll see how that can be used now let's ask it a question that will require the search ability what is the weather in SF currently we can see that it decides to use a tool and so importantly it lets us know that it's deciding to use a tool and then it also lets us know what the result of the tool is and then it starts streaming back the response so this is streaming not just tokens but also these intermediate steps which provide really good visibility into what is going on we can see here we can see the response that we got back from tavil um and then we can see um the response from the AI and so there's lots of dad jokes in here this is using open aai tool calling under the hood so we can also ask it to look up multiple things and see multiple tool calls in parallel what about in LA and in New York so we can see that it now calls two things two things it C it call calls the tavil search Json for weather in Los Angeles the tavil search for weather in New York we can look at the responses this is for Los Angeles this is for New York um and then we can uh see the response here so this is an example of parallel tool calling that's enabled with open ai's most recent feature their their tool calling so this is the assistant it's using Tools in an arbitrary way to accomplish its task let's uh go create a new bot and now let's create one uh of type of rag so rag is really focused on retrieval over arbitrary files that you can upload so you can upload files and then the uh you can also give it custom instructions um and and then the bot will respond based on those files what is the difference between this and the assistant because in the assistant you could also upload files and you could choose retrieval as one of many to tools the main difference is that this is much more focused on answering questions specifically about files that you upload so this means that it will always look things up in a retriever it's actually hardcoded and we'll show this in in the um when we talk about the the back end but it's actually hardcoded that it will always do a retrieval step here the assistant decides whether to do a retrieval step or not sometimes this is good sometimes this is bad sometimes it you don't need to do a retrieval step when I said hi it didn't need to call it tool um but other times you know the the llm might mess up and not realize that it needs to do a retrieval step and so the rag bot will always do a retrieval step so it's more focused there because this is also a simpler architecture so it's always doing a retrieval Step at the start and then it's actually always responding after that it's not doing potentially two retrieval steps it's not doing an iterative search this is a very simple rag architecture um which has its downsides it's not as flexible it can't handle multihop questions things like that but it's much more focused streamlined and that means it can work with simpler models as well so we have actually enabled mraw um through fireworks to work on uh this type of gbt a rag gbt so I have a uh I have a PDF here that I'm going to upload um this is Spade this is a paper Shrea recently wrote um at Berkeley and it goes over setting up kind of like a testing pipeline um Bas for for your prompts super interesting paper um I'd uh I'd recommend reading it regardless um let's upload the PDF um we can we can change the message slightly um let's still use 235 turbo research Spade let's save this now it's taking a little bit longer to save because what's going on under the hood is that it's injesting the file now it's in a method where it can be retrieves and and I'll talk about this when we talk about the back end as well um if let's take a look at this paper um and let's figure out something we can um ask what is a propped Delta so here it always uses retrieval the the it calls the retrieval function it gets back documents we format documents nicely um and you can see what these documents are and then it responds here so this is an example of a simple rag bot which always does retrieval hyperfocused on rag if you want to ground a bot in some external data source that you can upload this is probably the simplest and most reliable way to do that again it's a little bit less trustworthy than uh than than the chat bot we have it as a separate type of Bot because it is simpler so that means that it can work with other models like mraw which is an open source model so it just provides more flexibility and that flexibility is the same reason we have a third type of Bot this chat bot this is just solely parameterized by the instructions so you can write out long complicated instructions for how it should behave you can give it a character and it can act like that again because this is simpler it can work with simpler models let's create an example chatbot we'll create one that responds like a pirate so you are a helpful pirate always respond in a pirate tone pirate save this hi and we get back a response and pirate so a lot of gpts in the GPT store are really just complicated system prompts and so for those you can create them using using this chatbot type the other a lot of the other gbts that I've seen at least are the rag style chat Bots where they're parameterized by a system prompt and then also um and then also a bunch of uh files that you can upload to give it information besides what it knows about and it can search over those and so these you know these are much simpler architectures than the assistant but for a majority of use cases they're actually completely fine the nice thing about assistance is you can do more complicated with things things with it and you can also equip it with arbitrary tools and so here there's a bunch of tools that we've enabled in the back end by default but you can easily add your own and and explore with those um and so that's part of the power of this platform being open source as well you can Fork it you can make it your own you can deploy it either to end users internal company users anything like that other things that I want to highlight in the front end um you can make Bots public this means that you can share links um you can see old conversations and jump back in um you can create new chats um when you're in a conversation you can click in here and see the bot that it is using um when you create a new chat you can look at the saved Bots that you have if you have any public Bots they'll be down here as well and yeah that's basically it for an overview of the front end I'm now going to switch to talk about some of the architecture of the back end which will be really helpful if you want to Fork this and make it your own so this is the openg gpts repo it's under the linkchain org um there's some instructions here there's a good read me on everything that's in here there's some Docker composed files for deploying it um there's some other uh uh files for environment variables the API docs things like that for this I really want to focus on the back end so we can take a look at what's going going on in here um and most of the logic here there's there's some requirements files most of the logic here is going to be in app and so we can see there's a bunch of different files here so there's a few things that I want to draw attention to first let's uh let's maybe look at agent types so when we talk about the assistant in the in the uh in the front end this is where these agent types are defined and so different assistants have different architectures that are going on behind the hood let's take a look at the open AI agent for example so important to note this is built on top of L graph so if you aren't familiar with L graph you should definitely go check it out it's a really easy way to build these types of cyclical agentic Frameworks so we have this open AI agent executor which takes in a list of tools an llm a system message and then a checkpoint um and we'll see how we use this so first um we're going to create basically this uh quote unquote agent and this agent is responsible for taking in messages and deciding what to do next so there's first a step where we format um the messages um and so we add a system message um and that's defined up here and then we pass in the rest of the messages the llm also then has access to the tools so we bind it with tools and then we combine it using this pipe syntax to get this agent we next Define a tool executor this is a class that is just does some minor boiler plate for calling tools um and then we start to define the different nodes of the graph so first we Define the function that determines whether to continue or not this is should continue it looks at the messages if there's no tool calls then it finishes if there is tool calls then it will continue and we'll see how we'll use this later on We Now define the node that calls the tools um so here uh we uh take in the list of messages if we we get the last messages we get the last message we know that it involves a function call um because otherwise we would have ended um we get all the tool calls so when there's multiple tool calls we get them all we then pass them in uh into here into the tool executor in a batch so it runs them in parallel um and then we append them to the the messages and we return the messages from this node so importantly this will use a message graph and so this means that every node in the graph should return a message or a list of message so here we return a list of messages the other node that we add the agent this is just this is basically an llm call the LM call returns a message so both of the nodes return messages we set the entry point to the agent so when anything comes in we go to the agent we then add a conditional Edge so after the agent is called we then check this should continue function if uh it says continue then we call the action node otherwise we finish um we add an edge so after we call the tools in the action node we go back to the agent we then compile it compiling we're passing in check pointer equals checkpoint this is basically a way to persist um the state of the graph so we're persisting all the messages that that happen so this is nice for a few reasons um the main immediate way that we are using it right now is we are saving it to reddis and then we're showing that in the front end so in the front end when you see that we save the chat history um those are pulling from reddis the way that is getting saved to redus we don't have separate functions saving everything we just pass in this checkpoint and it all kind of gets written there there's similar things for the Google agent Google agent looks very similar there's some minor differences uh because it uh uh is a a Google agent so it's a little bit different it doesn't have tool calling it has function calling there's also an XML agent designed to work with anthropic models and so this is different as well so it uses some of the prompts um and things like that so those are the agent types um we also have a uh really simple executor for the chatbot chatbot just calls the message once with the system message so it has a really simple node the chatbot node just calls it and then ends dead simple um but we use the message graph again uh so that it it all of these Bots can speak kind of like on the same kind of like State um which will make it nice if we want to do any multi-agent or multibot things in the future um and then we also have this retrieval bot so this retrieval bot um basically it it's it had it's simpler than the agent node um so it doesn't have any Loops but it's more complex than the chatbot node so we have this prompt template um this is used for coming up with a search query to pass to the retrieval um and so we can see that we have the conversation here and then we generate the search query um and then the response pop template takes in instructions and then has context so what's going on here is that we have this get messages function um and basically what's going to happen is we're passing all the state around as messages um and so part of that um has the the search query involved and so we can see here if we scroll down I'm going to scroll down to this graph we F we have this invoke retrieval uh node we have this retrieve node and then we have this response node and then we always invoke retrieval at the start and then we go from invoke retrieval to retrieve and then we go from retrieve to response and then we end so remember how I said the difference one of the differences with the retrieval with the rag bot was that it always did retrieval this is this is what's happening so the invoke retrieval node it's always going to return an AI message that calls retrieval so we're not even actually calling the language model sometimes so if the length of the messages is one this means if it's the first message in the conversation we're just going to look up whatever that first thing was um so this is a little cheat that a lot of rag based systems or conversational rag Bas systems do is the first time someone types in something in we just look up that input the issue start starts to happen when you have a conversation so if I have a follow-up question or you know a series of follow-up questions I don't really just want to pass that follow-up question in because it could be referencing things previously and so what I do instead is I call um this other method um which is itself a call to a language model um and so this is using the search prompt to generate a search query um and then I specify that as the the retrieval thing then the retrieval thing is just calling the retrieval it's passing the results in this function message um and then the response is just a call to the language model with some formatting into this prompt so if we see this get messages thing what we're doing is we're getting the uh most recent message which is the result of calling this tool we know that it will always be that because we have this this determined graph we're getting the response from that and we're we're formatting that into the system message um and yeah so basically Al we're constructing the chat history the chat history is going to be AI messages that do not have function calls if they do have function calls then they are the result of retrieval steps and we don't want to include those in the messages that we pass to the final LM um and then uh the chat history also includes human messages um and then it also includes the system message system message is where we use the system message prompt as well as the context that we retrieve from the documents so that is the so those are the three types of um Bots that we have they're all put together in this agent file and this is where we also start to use um uh uh configuration basically so configurable fields and configurable alternatives are something that exists in linkchain and they're really handy when you want end users to be able to configure things or sometimes when you want to do the configuration on the Fly for example if you want to randomly select a model to use and you want to configure that on runtime and basically the way that that looks like and the way that we've implemented it here um is that we have this idea of like a configurable agent that wraps around a binding and there's these different parameters on here inside the initialization we take in the parameters and we construct the agent um and then we pass it in and then what we'll do down below and I'll return to this later on what we do about down below is we initialize this configurable agent and then we Mark certain Fields as configurable so agent field here is configurable with an agent type um the system message field is configurable with a system message um the assistant ID this is the assistant ID of the um of the bot that you've selected the tools are configurable the retrieval descriptions configurable and so a lot of the uh fields that we create the bot with are configurable and those are exactly what we expose in the front end of uh open gpts we've also exposed some configurable Alternatives and these are the different architectures so uh there's um there's a chatbot which follows the chatbot architecture and there's the chat retrieval which follows the rag bot architecture and if we scroll up we can see that chat retrieval uses this idea of configurable retrieval um so this is the same kind of like runable binding it's got these same parameters we Mark these fields as configurable um and then chatbots exactly the same and so basically the difference between the fields these are things that go in again we have like three seate types of high Lev Bots the configurable fields are things to configure this specific the the assistant type of Bot and then the alternatives are completely different kind of like Alternatives that you could even use so there's two different ways that you can configure things fields and Alternatives one more thing I want to highlight is just the ingestion bit um so there's the ingest pipeline um which is really quite simple this is something we're going to look to expand on in the future um and so if if you want to help make this uh retrieval more advanced would love that but basically we just split documents um and then we add them to a simple Vector store um and then the retrieval um is here part of tools I believe um and we can see here that it's just a uh retriever um really simple um really simple and then we just filter based on the name space so uh based on the assistant that you're using the assistant only has access to the files that were uploaded to it um so there's a lot that we can do to improve this we can add a reranking step um there's already some sort of query transformation going on based on the Bots um but we can add more things um and so improving the retrieval is one aspect that we want to lean into in the future um the last thing I want to point out is that this all integrates with Lang Smith so if you're a little bit lost about what the different types of agents are what exactly is happening uh uh what got configured um that's totally normal these applications start to get really really complex and that's where lsmith comes in handy so the deployed version we have hooked up to a project in lsmith and so we can click on here we have this open gpts example project um and so if I go in here and I click on a trace I can see exactly what's kind of like going on under the hood um and so here um this is if you remember this is actually the system message that I added um when I configured the pirate chatbot um and so this is a system message this is what I said and is the response and so I can track it and so I can also leave when I when I left the thumbs up and thumbs down um I can track that here um and so I believe yeah so here I left the thumbs up on this was this was the weatherman um I I got back this response if we click in what's here I think yeah so this is when we gave it access to the Search tool um and so you can see like exactly what's going on um this is a pretty simple agent because it just responded um but I can see the feedback here as well it's at the top level so I can see the feedback here as well I have a user score of one um and so yeah Lang Smith is a whole separate concept but I just want to point out that openg gpts if you deploy it it's integrated with Lang Smith um if you need Lang Smith access shoot me a DM on Twitter or LinkedIn and can get you access to that that's pretty much it for what I wanted to cover um hopefully this gives you a good sense of both how to use the front end the the research example as well as how to configure the back end um so I think I think the important thing to note that I would highlight is that you can put any different type of architecture behind here right now all the architectures the three different architectures we have the assistant architecture the rag architecture and the chatbot architecture they all use this message graph implementation which passes around a list of messages and I really like this because it's a single common uh kind of like interface that'll make it easy to add on different bots in the future um so there's a lot of things we want to do in the future one of those is having like multiple bots on the same thread or allowing you to switch Bots between threads um and so having this common state representation will make it easy to do that that's pretty much all I got hope you guys enjoyed this we're really excited about open gpts if you want to use this I mean one feel free to Fork it but also feel free to reach out to us we're more than happy to help thanks