Streaming Events: Introducing a new `stream_events` method

Key Takeaways

streaming is uh an incredibly important ux consideration for building L Ms in a few ways first of all even if you're just working with a single llm call it can often take a while and you might want to stream individual tokens to the user so they can see what's happening as the llm responds second of all a lot of the things that we build in the laying chain are more complicated chains or agents and so being able to stream the intermediate steps what tool are being called what the input to those tools are what the output to those tools are um streaming those things is really important for a nice ux as well so we've recently added a new method in Lang chain to help with this called stream events um which hooks into the Callback handlers we have so it's very easily modifiable and what it will do is it makes it easy to stream everything basically and so this video will cover uh stream events so we'll also cover the basic stream and a stream methods that we have so these are simpler methods for dealing with the end responses of of chains or of models and it will take a look at doing this for generic rentables and then for uh link chain agents as well so let's jump into it this is the docs right here little known fact all of these docs are also notebooks so I'm going to open up the notebooks version of this um let's restart my kernel um just so I can do it with a clean end um and yeah we're going to cover two main things here we're going to cover the stream and then a stream methods and these are methods for streaming the final outputs of chains and then we're going to cover the stream events um method we're going to cover the stream events method and this is a way to stream all the events that happen within a chain so um all runnable objects in link chain and so these are every these are all chains these are all models these are basically all objects in link chain EX Expos a sync method called stream and an async variant called a stream and so these methods will stream the final output in chunks yielding each chunk as soon as it's available and so as we see as we'll go along streaming is only available if for if every step in the program kind of like knows how to stream things along so if you're working with one step in the program like an llm if it knows how to stream then you can use stream if you're working with multiple maybe one of them doesn't know how to stream and we'll see an example of this later on then it might not stream so let's start by looking at the llms themselves so we'll use an anthropic model here and basically what we'll do is we'll iterate over the chunks in uh model.am things that come out of these are going to be AI messages um and so uh what we're going to do is we're going to append them to a list and then we're going to print um chunk got do content um and so this is just a Content field on the aage and we're going to print this little operator to to show this the spaces between the streams so we can see that it starts to stream as it responds if we look at one of these chunks we can see as AI message these AI messages we've we've made kind of like addable under the hood so you can add them up and you can get the final response okay so that's a simple llm now we're going to look at chains and so in this chain and so chain is kind of like anything constructed in L chain expression language this chain will just be a really simple one where we have a prompt a parser um and then the chain is prompt model parser um we will invoke it uh with a stream so we'll use async here and we'll stream over things and we can see what happens as it goes through and so basically we're asking we have a prompt that's formatting it with uh uh it's tell me a joke about blah um we have this model and then we have this parser and this parser is just responsible for taking the cont content property off of the message and so it just return to string instead of this AI message the model and the pars Expos stream method so we'll see that it will stream out as it completes there we go all right so we can also see an example of other things um so uh a more complicated parer that we have um is the Json output parser which takes in a response from a language model that is supposed to be Json um but it might be like half completed Json right so this happens when you're streaming the the like when you when you're halfway through the stream of a Json blob it's not valid Json and so this parser deals with a lot of those peculiarities and parses it accordingly and so we can see here that it will start to stream out the more completed kind of like Json blob as we create it here we'll see an example um where this doesn't stream um so this is a method that just returns all the country names um and so here if we add this on the end of The Blob we can see that there's actually no streaming that will happen because this final thing does it it doesn't have it doesn't yield things it doesn't stream it doesn't take in it doesn't take in a a generator and it doesn't return a generator and so it kind of blocks everything so it just returns it once at the end if we want to change that um we can change this and this will start to yield things and so why is this happening um I think this is happening because I have a uh I have some flush issues here um let's do this see if this changes things yeah so we can see here once I add in this so there's some flush issues going on but basically we can see that it streams back um tokens and again that's because this yields things com some built-in components like retrievers these don't stream anything so if I call uh the um if I call this it just returns a list of documents we haven't really seen a need for retrievers to be streaming so that's why we kind of haven't implemented a version of that but you can still use these within chains um that uh uh you can still use retrievers within chains and still stream the final results so here's like a really simple rag chain that gets the context back um so basically calls a retriever gets some documents passes it to a prompt passes it to a model passes it to an output pars here these still stream so we'll see that the final response we can stream all right so those are covering kind of like the basics of stream and aam the the big uh kind of like limitation of these is that it only streams the final kind of like output of a of a chain often times we might want to stream the the intermediate steps or the events that happen within the chain along the way so we'll walk through this new method um stream events um and the events that it produces are aligned with the Callback handlers that we have and so we have a bunch of different types here that we've listed here um so we kind of like have uh uh so this is for chat models and llms very similar we we yield something on the start we we yield something on the end and then on stream we also yield each token if the model support streaming same for llms um same for chains same for tools same for retrievers and then same for prompts prompts just has start and end um one thing to note is that this is a beta API um and would love to hear any feedback about it so please let us know so here we can start we can use the the we can just use the um model mod and we can we can call a simple model and we can take a look at the stream events that are associated with this model one thing that you may notice is this version thing in a stream events so what is this uh basically we're documenting kind of like the API for this a stream events um interface as mentioned it's a beta interface it may change so we want to uh make it make it easier to do that in the future let's take a look at what some of these events are so we can see that the first one is on chat Model start so it starts kind of like the the streaming the first event it hits the chat Model start and then it's chat model stream chat model Stream So basically what's happening is the input's going in it's starting the chat model and then after that it's getting token by token if we look at the last results we can see that it's on chat model stream and then on chat model end so this is a very simple single call to a chat model that first calls the event on chat Model start then calls stream on each token and then calls on chat model at the end let's take a look at the chain that we have so now we have this model and we have this Json output puror let's gather the list of events and we can explore them this way so the first three events that we see are actually all on blank start so first we have on chain start because we enter the overall chain here then we have on chat Model start because that's the first one but then the model streams things through so it passes a generator through it passes that generator through right away to the output parser and so we immediately hit kind of like on parser start we can explore this a little bit more by looking at the by looking at some different events so let's look at maybe like the last three events in here we can see that we have on chat model end on parser end and then on chain end um so this is this this makes sense it's uh you know if we imagine kind of like the the SE quence of of chain encapsulating a model and then a parser basically the model will kind of like finish then the parser will finish and then the chain will finish let's look at maybe some of the middle ones so if we look at like 3 to six we can see that what happens maybe let's do 3 to seven for just a slightly nicer thing um we can see that it's a mingling of of on chat model stream on Pur stream on chat model stream um sorry on chat model stream on par of stream on chain stream so what's happening is as the chunks occur because we stream them all the way through we first see one token come out of the chat model that gets passed through to the parser that gets passed through to the end of the chain and so it's the sequence of events so now we can take a look at uh using these real time to do some streaming so we will do this by parsing the revent the the the events that get emitted by the stream events so first we'll take a look at each event and we'll take a look at what the kind of the event is then we can basically do things depending on what the kind is and we'll just do some simple string matching based on the name so if the chat if it's on chat model stream this is when it's printing out a token we'll P we'll print this chat model chunk and then if it's on parser stream this is when it's uh going through the output parser we will pass that and we'll set flush equals true this will get rid of some of the flushing issues that I had above and yeah we'll we'll just only do this for the first 30 so we can see now that we start printing things out and it's intermingled with chap model and parser because basically the chat model starts accumulating tokens and at some point then the parser emits something um but but they start streaming kind of like at the same time there might often be a lot of different events coming out from your chains when you have complicated ones with more types of models um um different types of tools subchains things like that and so we want to make it really easy to basically um filter these events we have a few ways of doing that one is by name so you can assign any runnable so again a runnable is just a model or a prompt or a chain or an output parser you can assign it with a run name and then you can filter um the events by include names so here we give uh the model the Run name model the output parser the Run name my parser we have include names only my parser and then if we print out events we will only get events related to the output parser because we're filtering out the ones with the Run name model if we switch it up and we go model then we only get ones associated with the model another way to filter is by type so we have the same chain here um but we're just going to uh change include types and so now include types we're going to set next to chat model this will get all chat models so if you want to get all chat models regardless of the name that you give them um you can you can get them this way tags are another way to do it tags are basically inherited from any child component so here we're going to give this whole chain the tags my chain and then we're going to stream all events and include tags my chain my chain with so the way that tags work it's not just events associated with this chain it's with any sub child components as well so if we stream it out we see that we start to get everything we get chain um we get onchain start on chat model on parser start in this case because we taged the whole chain with my chain this is everything if we take a look at non-streaming components again we can see that so this is the this is the one that blocks the final result um doesn't stream if we look at a stream again we this is the same example as above it doesn't stream but if we do stream events it does and because that's get they get passed through up until that point so we can see kind of like everything going on another thing this is a slightly more advanced one is basically if you have um if you have basically custom tools and they have inside them a runnable um then what you want to do is you want to propagate the call back so that otherwise no stream events will be generated a common example of this is if you have a tool for an agent and that tool calls an llm you need to make sure to propagate the the callbacks correctly um so here is an example of it not propagating correctly um and so you can see that there is basically just the stuff for tools because even though it calls this runnable inside of the tool the callbacks aren't propagated so it doesn't actually know that exists if we now propagate them and you can do that by just adding in callbacks as an argument to this tool function um and then passing it in through callbacks here um you can see now that we get this onchain start and onchain end event and these occur in the middle and so these are I think more examples of just the same all right so that covers basic streaming stuff we get most of the questions around streaming around agents and so I want to show two examples of doing that so this is the streaming page in the Lang chain documentation that shows how to stream with the agent executor so let's restart the kernel so we can see what's going on here if you haven't if you don't know what agents are check them out in a separate video I'll link to one in the description basically with agents the first thing we're going to do is we're going to define a model and we're going to make sure that streaming set equal to true this is so that it's this is necessary so that it it streams no matter where it's called from so in an agent it will be called within the agent many times so we set streaming equals to True um and then we Define our tools um and so these are just two example tools tools um we can play around with them and then we initialize the agent and so here we can see that we are going to first pull a prompt and we're going to use this prompt for agent we're then going to list out the tools we're then going to create the open AA tools agent and we're going to give the model here a tag agent llm and then we're going to create the agent executor this is the runtime for the agent um and we're going to give it run name agent this covers streaming so this is the streaming of the agent the streaming of the agent does the individual steps so that's nice but often times we want to get the individual tokens as well so let's stream let's go down to custom streaming with events so here we can do the same thing that we had before so we have the input we'll use version one we're using a stream events and now we can start doing stuff with the uh the events that are emitted so if onchain start and if the event name was agent so this is basically this is if you remember back up here we tagged the agent executor with the Run name agent so this is basically saying whenever on on the start and end of this we are going to print out this and then same thing on the end of this we're going to print out this and this is needed because there's a lot of subchains within this agent um there's places where we kind of like do the prompt into the model um those are subchains we really care about this overall agent and then we care about the streaming of the tokens from the chat model so we are going to uh basically if it's on chat model stream and if the content is exists and so there's actually cases where the content doesn't exist and this is when tools may be called we can we can play around with this and see what that looks like after but for now only if there's content we're going to stream that and then we'll also stream the tool start and Tool end so if we do this we can see that it starts printing things out you can see that it really quickly streamed that um and then gets that so if we take a closer look we can see that it prints out starting agent we then get a bunch of information about the tool um tool one tool two and then it starts uh streaming the final response we can also change this so that it always prints out the content and so now what we'll see yeah so we'll see here so these things are when it actually so okay so this is when the content didn't exist we can maybe even just print out what exactly it was yeah here we can see that it prints out the whole chunk and so we can see that the tool calls um and it's still streaming kind of like stuff um which is why it looks incomplete but basically this is printing out the um individual chunks which have the chunks of the tool call in it so there's no content but there's the chunks of the tool call and that's what we're printing out here that covers streaming with agent executor there's one more thing I want to go over which is streaming with L graph L graph is a package that we introduced that makes it really easy to create uh agent run times as graphs and or state machines essentially the same thing and so because Lan graph is built on top of Lane chain and is uh runnable at the end of the day it has this exact same interface so if we go over to L graph we can see in example of this um let's also restart the kernel here we can create our tools we can create our tool executor which just runs tools we can create the model again we set streaming equals to True um for for chat open AI um we bind the functions to the model so it knows which tool it has available we're now creating the nodes in lingraph if you aren't familiar with this I'll also link to a video for lra so you should check that out we just did a whole series of them on on YouTube We Define a bunch of our nodes here uh um again we'll cover this in a video separately or it's already been covered in a separate video we Define our graph here um and then this app that we get back this is a runnable like any other thing that's constructed with Lang chain expression language so we can start taking a look at it and uh uh using it in the same way so we'll use a stream events we'll we'll look at the type of the event if it's a chat model Stream So if it is a uh token basically we'll do the same thing where we print it out if it's not empty um and then we also print out the on tool start and the on tool end so we'll we'll run this with the inputs what is the weather and SF we can see that we get back or it's logged the the starting input for the tool we get the output and it start streaming back the response um of the tokens there that's basically all I wanted to cover in this video streaming is super important to llm applications and so hopefully this new stream event method will make it really easy to stream back what is going on inside your applications to the end user again this is in beta so please let us know if you have any feedback or questions thank you