Build OpenClaw-RL + VoiceAgents using Claude Code | LLM context engineering series | Lecture 10

Key Takeaways

I'm discussing with you about two projects which I have implemented. And one of them goes slightly beyond the context engineering into a very unique space. And the second one is right related to what all you have been learning so far. The reason I have selected this first project is because from Raj and Sridhar in the last nine lectures you have been hearing about context engineering and uh how to design the right context so as to optimize your AI agents for your use cases. Uh so this this case study or this project is slightly different than that. Uh but I will I will try to make that link with context engineering and this project. And and the second project is related to voice agents. Which is something we have internally developed at VisualEra. So it is based on a very practical hands-on experience. So initially I want to start with the first project because I want to take your minds in a different zone. And I'm completely aware that the concepts in this project you might not be completely aware about, but I will try to make it as uh simple to understand as possible. So the way I have planned it is that I have I will spend the first half an hour on the first project and I will spend the next half an hour on the second project and then we will take a question and answers. So let me begin by sharing my screen and we will get started. Okay. Uh I hope you are able to see my Miro board. Okay, great. Uh Okay. So the first project that we will be discussing as a part of this workshop is called as Open Claw RL. Uh just give me a couple of minutes. Okay. Let us get started. As I mentioned we will be covering two main projects in this uh in this talk and uh the way we have scheduled this is that I'll be covering the first project in the first half an hour and the second project in the next half an hour. So there is a lot of information which I plan to distill in in this one hour of lecture. Okay. So uh the first project which I will be discussing about is called as Open Claw RL. And uh this is very specific. It is not completely relevant to context engineering. It goes slightly beyond that. But I will explain the way we have designed this and why it becomes very interesting. So if you look at uh the way Open Claw is designed, what happens is that there is an LLM which sits somewhere at the center. And a user is So let's say this is Open Claw. And uh this is the user somewhere here. And then what happens is that the user asks something to Open Claw. The Open Claw queries the message back to the LLM and the LLM gives us the answer back to Open Claw. And then Open Claw retrieves the answer and responds back to us. So in this whole process we are heavily reliant on the large language model which is fixed. We are not updating the large language model as such. So the large language model is frozen. All this middle layer of Open Claw is doing is it is acting as an interface between the user and the LLM. So if you were to ask me the question that okay Rajit, this this is fine but does this mean does this personalized assistant is it same for every user? Uh if if that is the case then why are people calling it as a personalized assistant if finally the same LLM is going to be used everywhere? Uh for all the users, for everyone in this call, let's say we are using Claude. So Claude is a LLM which is commonly used for all the users. Then where does the personalization exactly come from? And the personalization mainly comes from the context engineering which Open Claw has done internally. Uh which means that it has a it has a memory layer. And this memory layer includes both long-term memory as well as uh short-term memory. So there are two types of memories which are included by Open Claw in their architecture. There is also file which is called as soul.md. So this is the name sounds intriguing but it's it's really simple. It is some something where the soul of the user resides. So uh the memory keeps on updating uh and the soul is fixed. This defines your personality. Now if you look at this entire architecture, okay? of Open Claw. The level of personalization which is dynamic, okay? Let's just focus on the dynamic level of personalization. It only comes from the memory. It does not come from the soul.md because soul.md is fixed. Ideally what I want is I want my personal assistant to adapt to my responses real time and change according to my behavior or my conversation with the agent. Now let me ask you another question. Let's say you're interacting with this agent on WhatsApp, okay? So let's say this is the WhatsApp interface. Where we have multiple blocks of user and uh Open Claw conversations. Okay, and then uh somewhere here you say that no, this is not what I asked. Okay, and then uh your assistant replies back saying that okay, let me change that. Uh this is one conversation. There are thousands of conversations users have with Open Claw like this. Now, what I am talking about in this lecture is is your assistant learning from this kind of feedback? Now, this feedback is something which is not stored in the memory. It's not stored in the memory. Not in the file at all. But, it is rather what we call as a dynamic real-time feedback which is coming from the user. So, this is a dynamic and a real-time feedback. And the question that we are asking is is your assistant storing this information somewhere? It is a part of the conversation history, but later if you go on to Open Claw, let's say there are thousands of these conversations. Is is the uh the bottleneck here is your memory file. How long can it get? You have a short-term memory. You have a long-term memory. That is fine. But, some of these memories even get stored and extracted in the memory, but is this a good solution? Do you want a personal agent like this which has a huge amount of memory? And this is not what is happening in Open Claw at all. So, uh literally people have hundreds and thousands of conversations every day with their agent. And a bulk of this information, which is a very high-value signal, is getting lost. It is not getting stored anywhere. And uh that is a signal which we can learn from, right? Which which the assistant has to improve. And the reason that is not happening is imagine an uh analogy where you have a glass box. Okay, this is a glass box. And the LLM is like a brain which is sitting inside this glass box. This glass blocks boxes impenetrable. You cannot penetrate inside. So, we cannot touch the LLM. All we are doing is we are playing with the context around the LLM. We are not essentially breaking in the glass door and modifying the brain of the LLM itself. Uh because the door has a key and uh it is locked. We we don't have access to the key. The model is fixed. We we cannot change it. This is the main problem which current personal assistants have. They adapt to user feedback only to some extent. They only store a critical information, but they are not really dynamic to or or they are not agile to user feedback which is coming in through the conversations. So, the question that we are asking today is can we adapt your personal agents and uh make them dynamic? Make them somehow responsive towards human feedback which is coming in through communication platforms like Slack, WhatsApp, Gmail, etc. Uh but largely through Slack, WhatsApp, Telegram where people are interacting with these agents. So, we want to use that signal and the idea is to uh penetrate through this brain. And we want to modify this brain itself. Another word for it is that we want to fine-tune this. So far, we have not taken this angle in context engineering at all. Rather, we have been working with a fixed LLM and we have been changing the context around it. What if you ask a completely different question in a completely different sphere that what if we can fine-tune this LLM? Okay, uh and let's let's start to think from scratch here. What does it mean that we want to fine-tune the LLM? Okay, so uh let's let's let's take an example. Let's say the LLM is a box here. Okay, the LLM generates an answer. The LLM generates an answer. And uh let us denote that by this. This is the answer which my LLM has generated. Okay, uh now what happens is that a user is seeing this answer. Okay, so let's let's draw the user block over here somewhere. A user looks at this answer. Takes this answer as an input. And what does the user do? The user generates a reward or a feedback, let's say. Let's let's call it a feedback. Let's not introduce reward right now. So, we will have another block for this, which is a reward block. Okay, so we have this reward or a feedback. And this is a implicit reward. Okay, so which means that uh you are not the user is not saying zero or one. Whenever I'm interacting with Open Claw on WhatsApp, my feedback is implicit, which is captured in sentences like this is great or no, this is not at all what I expected. And then, ideally what I want is I want this reward to flow back to the LLM to update the weights. Okay, so I want to update the weights of the LLM based on this reward. Now, this is the basic flow which I want uh my system to have. And all of this we are understanding based on our intuition. We have not introduced any theory or any mathematics yet so far. Okay, now that we understand this, the next step is we cannot use LLMs which are gated. Okay, we cannot use LLMs which we cannot tune. Uh so, which means that we cannot use LLMs which are hosted on someone else's server. If you want to fine-tune these LLMs, probably the best idea is to have our own machinery, host the LLM there, and then update the weights. Okay, so this is this is the first insight uh which we get that okay, we want to implement this process, but we cannot make an API call now because that's not what we are doing. We want to fine-tune the LLM. And to fine-tune the LLM, you need access to the weights of the LLM. So, we need a different machinery for this. And we cannot use the existing API calls. Now, uh and we also need a method to update these weights. It's It's very simply I have written that okay, fine, you just update the weights, but what is the method for this? How do you actually go ahead, look at the reward, and update the weights? So, let's start with the second question first. It turns out that this this process, this mechanism has been playing out in the history of AI since the last I would say 40 years. And there is a name for this field which is called as reinforcement learning. So, learning from reward and taking actions based on the rewards is something not unique. People have been applying it to different types of uh problems. And now we want to apply this to LLMs. Okay, so we want to apply RL to LLMs. And uh application of RL in LLMs is something which is not that far back. It It It all started with uh reinforcement learning through human feedback, which is RLHF. And in fact, ChatGPT, which came out in 2020 to December, one of the main components there was reinforcement learning through human feedback. Okay, uh so so the the mechanism already exists. We already know the mechanism for achieving this. And there are algorithms in place to do that. This lecture is not about RL, so I will not be focusing on the on on the algorithms, but I want to give you a layout to you so that you understand where RL fits into this properly. So, the different mechanisms which people use are there is a mechanism I'm using mechanism, but it's it's called an algorithm. It's it's PPO. Then there is another algorithm which is called as GRPO. And for OpenClaw RL, we are predominantly going to use GRPO. GRPO was was popularized by DeepSeek, and since then many RL papers are defaulting to using group relative policy optimization. But essentially, the main outcome of this discussion is we have a mechanism for updating these weights. Now, we need a mechanism to first of all host the LLM. We need a mechanism for this feedback. And we need a mechanism for generating these answers. So, we will talk about three different kinds of mechanisms that we need. Uh first one, let us talk about the LLM itself. Now, in this example, in this project, we have taken the example of Gwen. You can feel free to take any examples. In fact, OpenClaw allows you to connect to local LLMs also, which means that you can download these LLMs on your PC. And uh you can do agentic tasks with them. But that generally requires more RAM because uh these LLMs usually have weights, and they they need a lot of memory in their KV cache. And a lot of people do not have laptops of that capacity. Uh so, even I did not host this locally, but rather I hosted it through RunPod. So, I I rented out a GPU uh using RunPod. And I downloaded the model from Hugging Face. And used my own machinery for the main LLM. Now, this is where our approach starts to differ from this. Here, the LLM is locked in a cage somewhere in OpenAI's servers, in Anthropic's servers, and you do not have access to that cage. But now, we don't need to use LLMs which are locked in cages. We can use our own LLM that we have the key to, and we can fine-tune those LLMs successfully. Uh that is exactly why this first step is is important, which is to use our own LLMs, and we are using the LLM which is Gwen for this. Uh so, the the LLM is where the responses are going to come from. So, now whenever you interact with OpenClaw on WhatsApp, you will not get the answer back from Cloud or GPT or Gemini, but rather you will get your answer back from Gwen. So, this is what the first step of our machinery has achieved, which is which is fine. We this is this is easy for us to understand. Now, the second step, which is generating answer, uh let me come to that after some time. First, let us focus on another important component of this machinery, which is how to give feedback. Okay? Now, generally in RL, in reinforcement learning, these rewards are numerical values. They assume values of zero, which means a very bad reward or bad performance, and one, which means good performance. But how do you generate reward for something like this? Now, we know that this is a feedback which is negative, so it should have a negative reward. And if a user says something like uh "Great, let's do that." This means that the user has liked whatever the agent has told it, and and this should get a positive reward, right? So, the magnitude or the value of the reward should change based on how the user is interacting with the agent. And the main question is how do you convert these subjective lines or these sentences which users are interacting with into a numerical value? And because we need to do that, we use another LLM, which is called as the process reward model. The process reward model is a second model which is used here, and the main purpose of this model is to generate feedback. Or rather, we can call it quantify the feedback. Quantify the feedback into a numerical reward. Okay, so this is the second model which is required for assigning these rewards. Now, uh in reinforcement learning, the two components we have uh readied in in place. And there are there are names in the literature for these two components. This first component, which is the LLM itself, it is also called as actor. Um I know these names are a bit weird, but this is called as an actor. And this is called as a reward model. So, it's it's it's a standard name for uh there there are outcome-based reward models, and there are process-based reward models as well. Okay, so we have two components now. One component is the main actor, which is generating the rollouts or the trajectories or the answers. And the second component is a model which is giving feedback uh to whatever the user is interacting with the agent. Okay? And and we we definitely need a model for that because these feedbacks are typically very subjective. Now, we want to establish a link here. We want to uh based on this feedback, we want to update these uh this LLM. That's That's our main task, and that is exactly what reinforcement learning helps us achieve. The algorithm that that we'll be using to achieve this is called as group relative policy optimization, and we will be continuously updating the LLM based on how [snorts] the user is interacting with it. Now, one question which you might have is uh okay, this is fine, but if we are updating the LLM, doesn't it mean that the OpenClaw is going to pause for some time before it gives the answer? How how can we do it fast? Okay, imagine that uh there is a brain which is used to give answer to the to the human. And someone opens the door in which this brain is kept with some screws and modifies the brain. Now, while someone is modifying the brain, the brain is not in a position to give the answer, right? Because its weights are actively being modified. So, it means that you have to wait before the brain gives you an answer, which is a significant drawback. Which is why what is done is both these processes are done separately. Now, what do I mean both these processes are done separately? The process of generating answers, which is this process, and the process of giving feedback, this process are these processes are decoupled from each other. Which means that the feedback is collected continuously. This is continuously happening as you are chatting with the um with the model, and after a bunch of feedbacks, so in this in this project, we are using 32, the model is modified. But while the model is being modified, the previous version of the model is also stored in the memory. Okay, so this this gets a little bit intricate here. So, now let's let's draw a chart of what all is there in the in in in my memory, okay? So, in my memory, the first thing is is you have the actor, which is the Gwen LLM, which is generating the answers for me. Then uh you have the reward model, which is the process reward model. Let's call it Lama. Now, again, what you have is you have another frozen copy of this LLM. Frozen copy of the LLM. So, this is being used to serve the answers while the actor is being modified after 32 conversations. Now, this is why reinforcement learning tasks are very highly memory-intensive because you're loading three models in the memory at at one time. And these three models are used to train your actor. And the actor becomes better and better with time. This this almost sounds magical, but when you see it working, there is a lot of satisfaction that you get in the process. Uh, we are essentially training the model. We are fine-tuning the model. We are changing the weights of the model in real time as we are collecting conversations from the user. So, this chatbot is actually personalized. And it is it is a more, uh, I would say granular personalization because I am taking into account every single interaction which the user is having with the, uh, chatbot. Now, uh, all of this, uh, process I have documented in, uh, one of the pods that we have created on Vizzuara and it will be very helpful for you to understand how this feedback mechanism works and how reinforcement learning actually works to transform this OpenClaw into something that is continuously improving with time. Uh, this is something which is very state-of-the-art. And, uh, this is, I think, the right place for people to get into RL also. For the longest time, reinforcement learning is considered by people to be very theoretical and they say things like, "No, this is for not for me. I don't understand maths, etc." But, the point is that you don't have to understand mathematics. You don't have to understand how GRPO works. As long as you know what the algorithm is doing, it is very intuitive because RL is nothing but learning from experience. We learn and we get better with time. Now, whatever I'm explaining, I have deliberately kept this whole process to be a black box for you because I don't want to intimidate all of you with what is GRPO, but it is very intuitive and a lot of fun to learn. Uh, okay. So, now, uh, I will not be, uh, just telling you the theory, but we have actually implemented this. And, uh, I'm I'm going to show you a GitHub repo in which all these details have been given. Okay. So, I'm I'm sharing this link in the chat in which I have given all the details of, uh, how to implement this. Okay. Now, whatever we have discussed, you'll be able to understand how to deploy this in in practice. So, as I said, you need a number of GPUs because you're serving three models. So, you can use either three to four GPUs. Uh, this will take at least 30 to 45 dollars of cost for a three to four hour session where you can actually see the model improving with your interactions with the model. We are using a Qwen 3 4 billion, uh, parameter model as the actor model which is used to serve the responses. And, uh, this is the flow. Okay. You're chatting with it and, uh, there is an actor which is getting trained. The process reward model is scoring the responses and the weights are getting updated and we are hoping that the next response is better and the model is continuously improving as you are giving responses. So, everything is specified very clearly here. You have to first run a RunPod account. You have to open up a pod and giving this container image is very important. Maybe I'll just explain to you how how that is done. So, when you go to RunPod, uh, RunPod is a website for people to rent out GPUs. Uh, go to this place which says H100. Go to H100 and select three GPUs. And now, this is where you need to make the main change. Click on the edit option over here. And you see here there is an option of container image. Now, in this option, you have to specify this. And there is an option here for start command and HTTP ports, which is this. HTTP ports is here and start command is here. That is all you have to do. And in fact, now all of you are have become experts at using Cloud Code, so you can just give Cloud Code this repo and ask it to do everything for you. Uh, so these thoughts should now come very naturally to you having done the entire course on context engineering without having me to tell you about it. But, everything is specified here in detail and, uh, I will show you how I am running it. So, you can see here, uh, this is my, uh, this is this is my local repo. Okay. And I have implemented this. And this is currently running live. You can see one GPU being active over here for RL. This is this is the one OpenClaw RL. And let's let's see. Okay. So, we have entered a live mode now. And Just a second. I I hope you are able to see my screen. Okay. So, I'm I'm assuming that you're able to see my screen. Uh, so this is the interface that you will see once you deploy that GitHub repo using Cloud Code. And here there are two options. Okay. There is also an option where you interact with through WhatsApp. But, right now for this demo, I'm just using this chat interface. So, let's let's Now, to help you understand what is happening, let me ask a question. What is a neural network? Now, I have asked this question. You see, there is it's thinking and GPU one is loaded. You see here, 83%. Now, this is the actor which is being loaded and it is generating a response. Qwen 3 4 billion parameter model is used here to generate a response. The response is now generated. Now, this is not coming through any API call, but I have hosted the model on, uh, this RunPod server. Okay. So, so so it becomes So, the answer is coming from that model itself. Now, I I don't like this answer. It's it's it's huge. So, I'll I'll tell it that, uh, I'll I'll give a feedback at the bottom. Uh, just a second. Okay. So, so the feedback I give is that this response is too verbose. Uh, keep it short and simple. And explain from first principles. So, now you see the, uh, GPU two is also loaded. And GPU two is the one, uh, which is actually the process reward model which is giving the rewards. Now, you see here, uh, it it it it has given a reward of it has given a reward of zero, which is which is what is expected, right? Because I I was actively telling it to that I don't like it. Keep it short and uh, it it gives it gets a reward for zero. Then then I say, uh, "Not great either. Give me multiple paragraphs." Okay. So, uh, now you will again see the PRM load up here. You see GPU two is loaded up, which is the process reward model. Again, gets a reward of zero because I was not happy with it. Now, let's let's let's do something positive. Okay. Now, I'll say, "Great. This is excellent. This is exactly what I wanted." Okay. So, now, uh, hopefully you will see a reward of plus one. Yeah. So, that is exactly what you see here. So, so you see the PRM is working perfectly. And at the top, you see the batch is progressing which says three out of 32. Which means that it will collect 32 feedbacks and only then it will update the model. So, this GPU zero which you see here, this GPU will only be activated when we are fine-tuning it. Right now, only the scores are, uh, collected and the scores will be used to update the model. Uh, Okay. So, this is this is exactly how how it works and how you can modify your existing OpenClaw assistant using reinforcement learning. This is this is a very practical tutorial which I wanted to share with all of you. And, uh, I know some parts of it I haven't covered because reinforcement learning I would have liked to go in more depth, but that is completely out of scope for, uh, this workshop. And I I hope, uh, yeah. So, Arun has asked a question that how do we know what weights need to be modified? That is a great question. And that question leads you to understanding RL in LLMs, reinforcement learning applied to LLMs. Uh, that is exactly what the theory of reinforcement learning mainly focuses on. Let me pause here for some time and take questions from all of you. Um people in the chat, please ask any questions. You want something to be clarified. Uh you want to understand how to deploy this in your own setting, etc. Please go ahead and ask the questions. How frequently weights are getting updated? So, at at the top you see here the batch progress says three out of 32. The The weights are getting updated after 32 rewards are being collected. And using GRPO, we will be fine-tuning when after 32 responses are collected. Is this application using RL to adjust the weights? Yes. So, uh this application is using reinforcement learning to adjust the weights of my main LLM, which is connected to Open-Claude. So, it is adjusting the weights of when 3 4 billion parameter model using reinforcement learning. Uh Shiva is asking that they want to replicate what I have done. Okay, let me again tell you the steps to replicate it. Um please go ahead and uh use this repo. I will give two answers to replicate this. The first simple answer simplest answer is use Claude code and interact with it to help you replicate this. The second answer is more detailed. The The first step what you have to do to replicate this is go to RunPod and uh click on a GPU provide uh let's say H100 X uh SXM. And in this example, I have used three GPUs. Remember, I had said we need three GPUs, one for the actor, one for the reward model, and uh one for the frozen reference model. And the important step here, which is something we don't usually do, is you have to go to edit over here. And you have to change the container image. So, now this container image is uh you have to give this container image of let's see what I have written. Yeah, slime RL slime latest. So, uh you would ideally type something like this. And you would give a starter command, which is mentioned here. And uh you would increase this container disk and uh the network the volume disk, also. And you would expose three ports here. Why do we need three ports? Uh there is one port on which the uh PRM is running, and there is one port on which the actor is running. So, we are using a framework called SGLang here for the inference, for the rollout. And we are using Ray for the distributed engine. So, uh these are the changes that you need to make, and then just click on uh set overrides over here. Set overrides. And then you just need to deploy on demand. Uh so, you will get one link, and uh you can share that link with Claude code, and it will help you set everything up. So, uh whenever you see this, right? There is There is a lot of satisfaction uh when you see the Open-Claude RL actually work, and the LLM modifies responses based on your feedback. That is why I wanted to share this with all of you. All these details have been mentioned in these uh pods, where there are a lot of Google Colab notebooks, which you can uh implement. I I find Colab notebooks an excellent way to understand concepts, because it helps us to visualize something in practice, which sometimes theory can't. So, yeah, to answer your question, this is exactly how you would go ahead and replicate it. Uh okay, let's see the questions. Okay, there are a lot of questions. Let's see. Okay, so uh let me summarize. There are a number of questions which have come up in the chat. Firstly, about the three GPUs, and what exactly are the three GPUs doing, and how how they are synchronized. So, uh you see here there are three GPUs, right? And GPU one is the one which is the actor, which is generating the responses. This is typically the one uh let's say you connect your Open-Claude with Claude uh or GPT. That is what this LLM refers to. That is hosted on this GPU. GPU two is where uh you're using a process reward model uh to score your responses. And GPU zero is the is a reference copy, uh which we are using to fine-tune. So, GPU zero will not be active unless 32 uh rewards have been collected. It will only become active after 32 rewards have been collected. So, uh to give you a terminology, the actor is the main LLM. The frozen reference model is is a different uh is is the same uh LLM, but but it's it's it's a copy of the actor. And then the process reward model is the third LLM, and that is what these three GPUs refer to. Uh the question is about how how does Open-Claude does the personalization? Open-Claude has multiple ways to do personalization, which it it uses the short-term uh and and a long-term memory. And it uses a soul.md file, also. But But as I said before, uh maintaining files for memory is is a good option if you are not a regular user of the agent, but if you have a lot of feedbacks coming in, then that will become a bottleneck. Because you you can only store so much information in in these memory files. Uh Yeah, so uh the reference model and the actor model are periodically synchronized. That is why their their their weights are coordinated. So, let's say uh you collect 32 rollouts, right? And you're using that rollouts to update the reference model. But at the same time, the actor model is being used to generate the next rollouts, and then the weights are being transferred every now and then. So, the reference model and the actor model, they sync at some time, then again they desync, then again they sync, then they desync. So, this this process continues. Uh So, that's that's the interesting part, right? So, it it it is uh the the the best part about this implementation is that there is no uh the model does not stop when you are interacting with it. And the reason is that uh while you are updating weights of of one model, the older version of the model is being used to serve the answers. So, that is the reason uh there is no lag whenever the user is interacting with it. So, this is this is the best part about the way uh you are doing the synchronization between uh collecting the feedback and updating the model itself. That is that is one of the tricks which which is which is a quite a common trick which is done in reinforcement learning. Because the same architecture you tend to see when even labs like OpenAI, etc., they fine-tune their models using reinforcement learning. Uh This is an approach that they use. Okay, so uh I can see that there is a lot of interest in the topic based on your uh responses. I would really encourage you to go through these pods after this lecture. And if people would like to get in the field of reinforcement learning, uh okay, so Rishabh has an interesting question. Uh can we have a single layer or a vector which captures the taste of the user? Yeah, so this is uh you can have vectors for each of the users. Isn't that something which Open-Claude already does? It It has a soul.md file, which is intended to capture the taste of the user. So, I I guess that is already being uh done uh uh with Open-Claude. Here, what we are doing is something radically different. We are changing the model itself based on the user's interaction. Okay, so now let's let's move on to a topic which is more directly related to the domain of context engineering. And uh this this topic is based on uh my own experience of building a voice agent. So, I I have created a PDF based on this. Now, this this title is given like that because that that's the title of my repo. I'll show you. This is the working repo. I've given it the title conversation agents for Bharat. So, for quite some time I was interested in voice agents and uh I'm I I I still am uh trying to find efficient approaches to build voice agents. Now, if you are new to the space of voice agents, there are a lot of uh model providers, there are a lot of middleware providers also. There are a lot of companies who are developing their own models for voice. India is also doing good there with Sarvam uh having their their uh their own voice models. The the main question which I wanted to address was I wanted to develop a voice agent for Vijaya. Uh using all our knowledge base and I wanted something which has a good latency and also the accuracy is decent. So, I I wanted to optimize on two parameters which is the accuracy parameter and the latency parameter as well. Now, uh because there is so much information and so many components available on the internet, it becomes overwhelming to set up a voice agent on your own. And that is where context engineering comes in very handy because if you know exactly what you want, you can completely ignore some categories and focus on only those categories that you are most interested in. So, if you directly prompt Cloud Code saying that I want to build an agent, help me build one, that's that's an approach for failure because you're not really uh telling it about, "Okay, what is specific to your application? What What are the metrics you are looking to optimize on?" And uh uh all all these nitty-gritties which you are uh aware of as as a uh as a person who has interests in in developing that voice agent. So, uh this is a document I've created to explain my journey through this process. So, I think it will help all of you uh who are envisioning to create voice agents. The first one is the knowledge curation and uh the knowledge base. Now, this this becomes extremely important because this is where uh everything is the responses from the voice agent will be based on this knowledge base, right? So, curating a good knowledge base and a tight knowledge base is very important. And we have been discussing this at Vijaya since the last 2 3 weeks now uh that an ideal knowledge base is something which is which is the right amount. Okay, it it it should not include everything. It it should ideally be very compact and with dense information. It is counterintuitive that Okay, you would think that if you feed in everything in the knowledge base, you would get an agent which responds well, but that doesn't work. Ideally, you need a dense knowledge base which is also very compact. So, the knowledge curation, there is a lot of time that needs to be spent by the user because that essentially becomes a major part of your context. So, uh we we looked at various knowledge sources. One was the emails that we get in our inbox. Uh and then the second one which is not really mentioned here is the YouTube transcripts of the launch videos that uh we make for all our bootcamps. So, that is a very good uh high-quality signal that we have for our knowledge base. And using that, I used Cloud Code to create a good tight knowledge base. And then I also uh asked it after the knowledge base was created, I wanted to test whether just based on the knowledge base, is it able to answer all the emails which are coming in our inbox? So, turned out that it's not the case. So, there were some iterations there when you are curating the knowledge base. So, that is that is the first step that all of you have to focus on whenever you are building any voice agent. Uh or maybe any agent, the knowledge curation is the most important step and you have to spend a lot of time in that. Then, the next step which I will come to later is uh RAG. Okay, so uh You you have an option of whether So, we have taken a separate dedicated lecture on RAG in this workshop. And but it turned out for me for this project, RAG was not the great option. And the reason it was not a great option is because the latency increased significantly whenever you are searching through embeddings and finding the most similar embedding. And finally, I ended up passing the entire knowledge base in the system prompt itself. And that worked brilliantly for me. It sounds very simple, right? You you might say that, "Okay, this is not good. RAG is an amazing uh uh method." But the the the elegance of the method doesn't matter. Okay, finally, your aim your final aim is that your metrics of accuracy and latency should be satisfied. Whatever algorithm you use to achieve that, that is fine. So, I I uh injected everything into the system prompt and it worked brilliantly for me. Uh Okay, so this is this is regarding the knowledge base itself. But then, you have to define a lot of tool definitions which is searching. So, so this is mainly relevant for RAG. You need to define a tool which can search through the knowledge base. I I also tried a method which is uh keyword matching which works better than RAG actually, which is also a bit counterintuitive. But because of the specific nature of our knowledge base, it turned out that the entire knowledge base was structured into a very nice PDF uh or uh rather very nice sections. So, the sections were segregated very nicely by their keywords and that's why the retrieval worked brilliantly using just keyword matching. And that was pretty fast as well. Okay, so uh the the first thing that I want to start off with is uh the voice pipeline itself. Okay, so I I started researching about this voice uh this voice pipeline and I found out that there are two ways of uh doing voice-to-voice conversion, okay? So, so this is a voice agent and I'm speaking through my voice and the answer should I should get back again uh should be a voice. The first option which is traditionally used is a cascading pipeline which is you convert speech-to-text, use an LLM, uh process that text, and then you convert the text back to speech. So, this is an uh STT LLM and TTS pipeline which you have to use to build a voice-to-voice agent. Uh and Okay, so this is the cascading pipeline. This is great. One question which I had was, "Can we do direct voice-to-voice by using audio tokens? Not using text tokens, but directly voice-to-voice." And it turns out that I've probably not included that here. There are options for that. Uh you have Gemini Live and you have GPT-4o Real-Time. So, experimented with that as well. GPT-4o Real-Time turned out to be excellent for my use case. The only problem with GPT-4o Real-Time is that it has a huge cost. So, this cascading pipelines, they they uh turn out to be good for reducing your cost. And along with the latency and accuracy target, I had a cost target also. I don't want users to interact with 100 minutes every day on on my platform and I have a huge bill to pay. Even though the accuracy, latency, everything is great in voice-to-voice, and uh I still think voice-to-voice modalities are the future for voice agents. Uh but the the cost is an order of magnitude higher than these cascading pipelines. Uh okay, so that is that is the first uh remark uh there which is about voice-to-voice and cascading pipeline. So, I experimented with both and uh I learned something pretty interesting. Now, there is another thing which none of these pipelines can do, which is uh understanding Indian accents uh better, understanding dialects, and Sarvam AI excels in that. But I faced issues with Sarvam AI and my hunch is that Sarvam has released APIs which are not up to the mark with with the APIs that they currently have developed in the lab. I don't know why that is the case, but they have deliberately released APIs which are maybe of earlier versions. If you try out their APIs, you will you will notice it immediately. There is it it it doesn't really perform great. Uh that was my experience using Sarvam. So, uh I I I did not go with it. Although, I was very excited because Bulbul was released at the same time when I was experimenting with this. So, I was expect uh I I was excited to try out their API, but but it didn't work well. Uh Okay, so let's let's go ahead. Okay, so the first is this email analysis and the knowledge base curation. Uh as as I said uh I implemented an email extraction pipeline. And we created a great knowledge base. Uh there is still scope for improvement here. I think. Uh and uh after I created this knowledge base, Raj created an even tighter knowledge base which reduced this knowledge base directly in half. And it started giving better responses, which I did not think was possible, but that that turned out to be the case. Then I tried this Google Gemini live API. Okay, so this this has a good cost. Uh which is which is comparable to cascading pipelines. This is also a native speech to speech model. But, the problem with this is it the latency is it is very slow in the response. Sometimes it just pauses and doesn't give you any response. So, I I would have actually preferred this if this would have worked, but it was extremely slow and I did not go ahead with it. As I explained, OpenAI real time was very high quality. And uh it it it performed well. But, uh and it has a very natural voice also. But, it is it is a lot expensive than than than other uh other cascading pipelines. Uh now uh I'll I'll explain about WAPI a little bit. Okay, so now with these cascading pipelines, if you look at there is a huge amount of literature and and it becomes very overwhelming to choose the right pipeline. So, I implemented an ablation study where I conducted a series of experiments. Uh if if the auto research by uh Karpathy would have been released at that time, I would have probably gone for that. So, I conducted six pipeline configurations. You can see I have used different STT, LLM, and TTS models. These are the state-of-the-art models. You can see there is Deep gram. There is Whisper. This is the Indian specialist which was Sarvam and Sarvam Bulbul. Uh there is uh Deep gram again. There is 11 Labs, OpenAI TTS, Cartesia Sonic. Grock Lama 3.3 for the LLMs, etc. Now, uh what I realized is that with the cascading uh strategy, every single component takes up some time. So, if you add up all these components, the latency increases a lot. Even though the quality is good, all of these are state-of-the-art models, but if you combine them together by yourself, it it increases the latency a lot. You don't want your agent to wait for 3 seconds before responsing, right? That is why I switched to uh this WAPI pipeline, which I found to be excellent. Uh this is There are other There is one more alternative to this. I'm I'm not able to recollect their name. Yeah, I think it's Reteller. So, these guys have a uh have an interface where you can select models for your cascading pipelines. But, their main USP is that they have minimized the latency. And and they have done it very well, in fact. The cost is also decent because finally it's a cascading pipeline. They just add a uh platform fee per minute. And uh it's it's out of all this was the best experience for me, and this is what I selected at the end. Because it had both the benefits of uh the low cost for a for a cascading pipeline. And it had a low latency also, so it it worked out uh pretty well for me. And within WAPI, I used Deep gram for STT and 11 Labs for TTS. So, uh the final solution turned out to be just directly using a platform, which which which sounds trivial, but there is a lot of research which has gone in before I arrived at at the final solution. So, this is the uh Sarvam full stack. Uh they they have done quite a lot of innovations. Uh but, I I did not find this to be to add a value compared to the WAPI architecture which I have developed, and the reason I have already mentioned. Uh I I really hope their API quality increases or uh uh I'm not sure, but basically uh they have a lot of advantage with respect to dialects. Uh WAPI or other platforms, the biggest thing is that you you get a speaker with a US accent, and they they modify it to make it sound like Indian. So, so it ends up sounding like Indian, but it sounds like an Indian American talking with you. It doesn't sound like an Indian who is brought up in India. That is the main advantage which uh Sarvam offers, and I really wanted that in my voice agent. So, I spent a lot of time tweaking this, but I was not able to get it right for an actual production level application. Then for the rag, as I already mentioned uh uh the knowledge base uh I I tried uh using rag, but the tool calling itself took a long time for me. I had this tool which is search knowledge base. And every time it uh I I gave a question which was out of the system prompt, it used that tool. It was intercepted, and the answer was given, but the latency was very high in in this whole process. Okay, so uh This is this is my final analysis. The The final cascading pipeline which worked out to be best for me was Deep gram. Uh then OpenAI Mini and uh and and 11 Labs as as TTS. So, Deep gram for STT, 11 Labs for TTS, and uh Mini for uh the LLM. And uh I am interested in these two also. Uh In reality, with the accuracy and latency, I'm the most satisfied with OpenAI real time. But, the only thing was it is significantly very costly. Otherwise, I would have picked that. And for my production pick, I choose uh WAPI for my application. Uh so so this is the place This was the application where I used a lot of uh context engineering uh uh concepts, mainly curating the right knowledge base. That is the first thing. Second thing is uh rag versus system prompt. That was a major decision for me. Third is the detailed ablation studies across multiple pipelines, STT, LLM, and and TTS as well. And we have actually deployed this on Vijuvara. You can go to the website and uh give your feedback there. Uh it is it is currently deployed. We have been getting a lot of feedback uh based on that. And it's working out to be quite well for us at the moment. Uh we have also developed a tool which automatically updates the knowledge base as and when we are updating the new course and a boot camp. So, uh this this was a very interesting experience for me. I might give a more detailed lecture on this sometime soon, but right now for all of you, I think uh this is very relevant. We will upload this PDF in the dashboard also. So, that all of you can understand our experience building voice agents. So, voice agents is a field which I am very passionate about, and I think uh having an agent which is which has low latency, high accuracy, and also low cost, you you have to do some ablations to get to that point. So, for me, this is something that that worked quite well. And finally, we even deployed it in uh production. Uh Are Are there any questions with respect to this? So, these are the two projects uh which I wanted to discuss today. Uh yes, there is a speech model Persona Plex. And uh there is also Quan TTS which is released. There is also uh uh Mistral uh TTS which is recently released. So, it will be interesting to compare uh these also along with the other models. Okay, everyone. So, uh this was the two projects I wanted to cover in in my lecture. My main objective was to help you understand how truly personalized assistants can be developed using RL. And hopefully, you got a lens into the field of reinforcement uh learning also. Uh And and uh the second one was the voice agent using the WAPI platform. And there actually, I have created an agent which you can call with a number also, but it it it is a US number. Uh so, we have not yet released it uh as as an Indian number yet. I would encourage all of you to try it out. Since I'm on Zoom right now, it might face some latency. But, if you go to the website you will see that there is this option of ask AI which is where the Vikrant agent will start speaking with you. There are two resources which I will share with all of you along with for the enrolled users on on their dashboard which is the PDF and also the pods link to the open cloud. Okay everyone, thank you so much for attending and all the best. I hope you use this concepts which we have taught in the context engineering to your advantage because people say that AI agents are the future. You can build applications, you can do things but unless people find value in it there is no use in there is no use basically unless some there is an end user who is using it and to have an end user you need to develop the context around your agent very properly and the only people I think Raj also discussed this last lecture. The people who thrive in this field at least I have seen who have who put a lot of care for their products. It's not like I have to just spin up an agent and deliver something but finally there should be a lot of care which should be implemented in all your products because people see that at the end. You just you don't want to just use your agent. For example, if I use directly use cloud code to build an agent it will give something but how do I know that's the best option out of all possible options? You need to do your own research for that. Cloud can help you do that research also but you need to sit around with it for a long time to do all these ablation studies. So the main purpose of this series was to give you a lot of grounding into context engineering and to tell you that agentic engineering is very powerful but it's also a skill which all of you should master and it should not be taken lightly but the product should be done with a lot of care and and effort. Okay guys, thank you everyone and all the best. Bye.