4 Types of APIs for LLM Apps (HTTP, Streams, Websocket)

Key Takeaways

Hi guys, welcome back to the channel. Today we'll talk about the four different types of API you can build for your LLM application. The first two are going to be variations of a HTT a classic HTTP API. One's going to be synchronous, so blocking, the other will be asynchronous. And then we're going to take a look at the more functional APIs for these kind of applications that uses either HTTP stream or websocket. So let's start with the first one which is going to be the blocking API or the synchronous API. That's going to be very simple. Let's say you have an endpoint or an API route uh called slashquest and the user can pass in the prompt in that uh in that route. When they do that, the backend server forwards the prompt to the to the OpenAI server. Let's say we wait 15 20 seconds depending on the complexity of the prompt. After that, the server forwards the response from the from OpenAI back to the client. So, you can see an example here. I'm going to zoom in a little bit more. Yeah, you can see an example here where the client is sending hello world and OpenAI is sending hi there back to the client through your server. So the biggest problem with this approach is the client does not get a response until open AAI processes the full prompt. Right? So let's say if you have a very complicated prompt and it takes 10 to 15 seconds for open AAI or any other LLM to form the complete response then the client is waiting for those 15 20 seconds. Maybe they see a loading spinner or just like a prompt that says loading, but the client cannot see the first couple of lines immediately uh like you typically see when you're using chat GPT claw or any of those apps. So even though this kind of an API works very well where in cases where you can immediately send the respond uh sorry send the response back to the client but most likely in cases where the prompt is uh much more complicated it will not work. So now let's build on this and then come up with a little bit of a better implementation which is going to be our asynchronous API. So in this one again we have a very similar looking route called slashquest which takes in a user prompt. Let's say the client sends a prompt to the backend server and then uh we forward that to open aai. So now in this design instead of waiting for openai to finish the response or finish forming the response we immediately send back a URL to the client. So we get the request from the client. We forward it to OpenAI and while OpenAI is processing the response, we immediately send a polling URL to the client and uh HTTP 202 status code. So what does this polling URL have? Right? So this going to be like a typical HTTP get route that you build in your backend server. And let's say it takes in a task ID, right? So whenever in this first step the backend forwards the response to OpenAI, it tags that prompt uh with that task ID, right? And then it sends back to the client that task ID and the full polling URL. So what the client can do in this third step over here every couple of seconds it can use that polling URL fill in the task ID and then ask our backend server uh something like hey is the response to this task ready or not. So the response when the task uh has been completed would be something like this where we give it the status the original prompt and the answer. So even though this is not a significant improvement over the first over the first design, it still gives the user a lot more feedback because now the user can uh do other things while the while the processing is happening in the background, right? So we're not blocking uh we're not blocking like we were in this API. Instead, we are setting the expectation immediately and telling the client, hey, this is going to take a little bit of a time. So, you can do other things and keep checking for a response and we're going to give it to you whenever it's available. So, totally depending on your application, you can build your client to pull for a response every couple of seconds, every 10 seconds, uh every second. That totally depends on your app. Okay. So, both of these uses the typical HTTP APIs that you're used to. Now the next two that we're going to take a look at are going to be HTTP stream and websocket. So both of these are going to be a lot more realistic to use for any LLM application. So let's start with the first one which is the HTTP stream. Okay. So uh for HTTP stream the cool thing is once the client sends the request to the back end instead of the backend sending only one response back it can send its response in chunks right so let's say your prompt uh hello world has uh three lines of response right now in your previous implementation we needed open AAI to send us all three lines of response before we and send that back to the client. But using stream, what you can do is uh you can tell OpenAI to send the response in chunk to your server and you can forward that back to the client incrementally. Okay? So you don't need to wait for the full response. So if you remember in the first case, we're assuming that the prompt is complicated enough where it takes 15 seconds for OpenAI to form uh its complete response. In the HTTP stream approach, we don't have to wait that long. So even if it does take OpenAI 15 seconds to process, we can start sending data back to the client as soon as like 1 second or 2 seconds. Okay. So an example if we go go through this diagram here it's going to be we get the request we send it to open AAI then immediately within like a second or two open AAI sends us the first bits of the response so for hello world it just says hi right and we forward it to the client maybe a couple of seconds later open AAI sends us there and we forward it to the client and at this point the client is saying hi there similarly we keep going maybe a couple of second Later it sends us how are. So now we have hi there, how are and finally you. So we have the full respond full response which is hi there, how are you? But instead of waiting 15 seconds to display to the user, we were showing the data to them in chunks which is significantly better. The one thing HTTP stream does not give us is the the chunking ability both ways, right? or the communication both ways. So in this case as you saw the server could almost push the data to the client but the client cannot keep pushing data to the server. So once the client fires this get request here even though it is getting the response in chunks until it finishes getting the full response it cannot just start sending another request to the server. So it almost limits that uh that uh chunking way of responding to only the server. The client cannot do that. So you can imagine um at this point let's say when uh when the when the server send there and we have hi there let's say the client decides to interrupt the server right and says hey no that's not what I was asking. I was actually asking this. In a case like that, the client cannot do that until OpenAI and your backend server complete sending the full response. So it needs to send, "Hi there, how are you?" And only after that can the client ask its second question. So now let's improve that, right? Which is going to be in our final design where we're going to use websocket. So using websocket, it gives you full control over the application. the the server can keep pushing data to the client whenever it needs to. Similarly, the client can interrupt the server at any point and just ask a new question. So, if you go through this example, uh it's going to be very similar to the last one. Let's say the prompt is hello world and now open AAI sends back hi there and then how are y and let's say before it can even complete by saying how are you the client has a new question right so in this case the client just wants to exit out of the app so it says got to go by now instead of so what happens here right so when when the client says got to go by the client doesn't need to wait for the first response to finish, right? Instead of that, the moment the client sends them this message, OpenAI gets that and immediately it starts responding to that new uh to that new prompt that the client sent. So in this case, sorry to see you leave so soon. Okay, so that's what you get with websocket. It's super powerful because you can interrupt each other at any time. And for applications like uh chat GPT or any similar application that uses LLM, websocket is definitely going to be the way to go. The only problem is it's always a lot more complicated to design something that uses websocket as opposed to HTTP stream. So if your current architecture uses HTTP stream, uh building the application using HTTP stream is going to be a much incremental uh change to your implementation than just switching to websocket altogether. Uh so yeah, hopefully this was helpful. I will attach the notes and the code in the description below. So if you want to go read through it, there's a lot more details there. Feel free to do so. And if you have any questions, just leave them in the comments below.