Make your agents 10x more reliable? Flow engineer 101

Key Takeaways

For people who build AI application before, you'll know that building AI application is somewhat weird experience. Everything works during test, but after release, things looks not as good. You just keep praying to God that please let large language model follow my prompt. The nature of large language model is that it is undeterministic. And this led to many weird and interesting prompts that people have tried for past 12 months. I remember seeing an awesome open source project where the author put part of prompt like this is very important to my career, please follow for some important instructions. And there also some common prompt tactics that people tried, like give GPT-4 some tips to making sure it follows some specific part of instructions. And believe it or not, some of those weird prompt actually backed up by some academic paper to prove that test actually works. But in general, one challenge across all the AI engineer is how can you tame the large language model better to making sure it follows instructions consistently. A lot of people are talking about how the next stage of prompt engineer is a flow engineer. Like Andrew Capsy talked about how flow engineer moving from prompt answer paradigm to a flow paradigm, where the answer is construct iteratively. And there also popular open source project like LangChain start building products specifically for this type of flow engineer design. And I think to many people, mostly you can understand the concept of flow engineer, which is instead of getting the large language model a prompt, we can break down a task into small steps and try to improve the result iteratively step by step. But even though conceptually it makes sense, there's not a lot of information about workflow engineer actually is and how do you actually adopt it in your day-to-day a large language model application design. That's why today I want to dive into a bit more about workflow engineer actually is and how can you start adopting those methodology in your design. So at high level, flow engineer is something in between large language model chain, which is extremely reliable but not very flexible, and a free form AI agent, which you basically just give high level goal it'll try to use whatever tool it has access to to complete task in a very dynamic way. But flow engineer is something in between. You would define a high level steps, but still using a general language model to control some part of the flow. And let's take a very concrete example. Let's say I'm building a research agent. If I'm using large language model chain, I'll probably build a very specific function flow where it would do the Google search based on a topic user give, and then it generate outline first, write each section, and merge them together, and call it down. So, this flow is extremely reliable and specific, but downside is that the maybe the research is not as good because commonly the research can take multiple different tries until you can find right information. Same as a content generation. But if you build agent to do the research, you don't really need to define a specific process. You just give agent list of a different tools and functions that it can have access to, like Google search, extract website content. And you just give a topic and it will be able to plan, think what kind of action it can take, and continuously taking all sorts of different action until it thinks there's no further action can be taken. So, benefits here is that it is extremely dynamic and flexible. The problem is often you don't really have control about what kind of process the agent should take. And if you ever try to build AI automation for a business, most of the time they actually do have a specific procedure process that AI needs to follow. And this where the flow engineers start becoming interesting. So, flow engineer is basically try to get human predefined what are the high level process or procedure that the AI should follow. But you can still get large language model to make a lot decisions in specific steps or points. Like you can do the research and have a check to see if all information has been found. If no, then do it again. And repeat this process until all information is found, then move to next step, which is write content, where you can also insert a critique and review process. And now you might argue that to achieve this type of process, theoretically, we can just give a agent a specific instruction and system prompt about a specific process it should follow, and that should already achieve some kind of flow control that we want to achieve with flow engineer. Well, that is true for many of the simple cases, but when you dive into a lot of real user cases, the procedure and SOP is a lot more complicated than a few simple step flowchart. It can have a wide range of different branch, and it is extremely difficult for you to convert a complex flowchart like this into a prompt. And even though you do, as the process became bigger and bigger, the agent starts struggling to follow instructions because there are limited amount of effective context window. So, just using prompt to guide agent to follow specific procedure might work well two times, but it's very hard for agents to consistently follow instructions. At this point, you might wondering, isn't this flow control looks extremely similar to those multi agent frameworks that people have been using like CrewAI or AutoGen. And that is exactly right. Most of those multi agent systems provide a way for you to control what are the different steps that AI should follow by breaking them down into different agents. And all those multi agent framework like CrewAI or AutoGen, they just provide different ways for you to control flow and different ways for you to manage memory and state. For example, if you're using AutoGen, the way the person flow is controlled is that they will have something called group chat manager who is continuously monitoring the chat conversation and decide which node or agent to go next. And state shared across all the agents are just the full or partial chat history. And that's the reason why when you run AutoGen, you'll realize the cost is a lot higher than just any normal agents because for every single point the large language model is processing the full chat history. While LangGraph is just another framework that allow you to control the flow and state management. You can basically control the flow using either code or large language model. And the shared state can be a bit more optimized where you can define specific important information like for a research agent example, you might just want to share what kind of website has been scraped already and what information we collected for each data. And if you're interested, here's my take of different sorts of multi agent frameworks. They all have its pros and cons and it's hard to say which specific one is the best. But the one I want to dive into a bit more today is LangGraph. From what I can see, LangGraph seems to be a framework that does provide a lot more flexibility in terms of flow control and state management. Though it is a lot more complex in terms of setup, I think it is definitely worth diving into. So, what is LangGraph and how does it work? Fundamentally, LangGraph has two key components. One is graph, which you can just consider as a flow. Another is a shared state. So, shared state is the memory or context that is going to be shared across all different steps. But before we get into it, I know many of you might be using GitHub Copilot, which was absolutely a game-changer. But recently, I canceled my GitHub Copilot subscription because I found something that is better and free. That is Codium. So, you can think of Codium as a free alternative to GitHub Copilot, where it has a killer use case of auto-complete but with a bunch of unique and useful feature. Like you can ask Codium to refactor your existing function to add a new feature, fix bug, or add comments and documents, or translate your Python code into other language like JavaScript or TypeScript. And one of the most powerful and useful feature to me is that Codium is context-aware, which means it'll be able to explain and answer your question about the whole code base. And this is extremely useful because when we start a new framework like LangGraph, often there's no enough documentation to tell us how to use this framework. Every time when I open new repo like LangGraph, if I open Codium, you can see that it start loading and vectorizing the whole code base. And once it is finished, I can go to chat and start asking any question about this code base to understand better. For example, when I'm viewing some example, I don't really understand what tool node is. So, I can just switch to Codium and just ask for what is tool node and do command enter. It'll start looking through the whole code base and explain to me what a tool node is and also the code example. So, this really speed up the learning process for me whenever I use a new framework. And this is for you to chat with AI in any sort of context like a specific function or class or even popular repo on the GitHub. And while I'm in the code editor, I can just do command I to activate Codium and ask help me build a React agent with LangGraph. Then it'll start processing and generate a whole code example for me about how to use LangGraph. And I can either accept, reject, or give feedback by follow-up. And for existing function, I can click on this refactor button and ask it to automatically add documentation and comments for me. And most important thing is that it is free forever if you are just individual user. They really just try to make a money for teams. I've put a link in the description below where you can go and click on get code in and select any type of code editor you are using now. If you're using Visual Studio Code like me, there's a quick button you can just click and install. Now, let's get back to what LangGraph is and how can you use it. So, firstly, let's talk about the key components of graph. There are two key components. One is called node and each node it you can think of as a specific step in your workflow. It can be either a large language model call or a specific function you can call or it can be even an agent. And second concept is age. Age is basic connection between different nodes so that you can use to control flow. If you are building a research agent, your graph might look something like this. You will have two node. One is a website research agent that can do the Google search and find the latest news or information. And the second node is a large language model call to generate report. And here will be a connection age between these two nodes. And this is one of the most simplest possible graph. But the age can also be conditional. For example, if you want to make this research agent a bit more powerful, we can change the graph a little bit. It will try to research within the company's website first to get a first-hand information. And after that, we can run conditional age where it can run a function or get a large language model to make a decision. Is the all information that we supposed to find already found? If yes, go generate report that I found. If no, we can pass on to another agent who have access to whole internet so that it can research and pass on all the research data to the large language model to generate report. But to achieve such workflow properly, we actually need some sort of shared knowledge between the first agent, second agent, as well as large language model step in the end. And that's where the second part state come in. So, the state is almost like a shared memory between all those different steps. So, in this research agent example, we might have a few different states. One is the website that we have script already as well as the latest data point that we found from different states. And those states can be updated and read from any single steps. A framework like AutoGen is pretty much similar. Except in AutoGen, the process of flow is controlled by the group chat manager, which is large language model that going to look at conversation history and decide what's the next step and actions. While the state is basically the full chat history that keep broadcasting to every single agents. Whereas a framework like LangGraph give you a specific information inside state instead of passing on the full conversation history, so you have more control of the memory and all foundation of the cost and the flow most of the time will be controlled by the code instead of just the large model. So, this is pretty much how can you build a reliable agentic system. It does require a bit more pre-planning because you are basically offloading a lot of planning and decision-making from the agent to the system designer, which is human. And the common process to start design such system is that you firstly want to map out the whole flow and graph, so that you can understand the key components. Then you can start listing out all the different states and data will be useful and necessary to be shared across every single steps. Yeah, I want to show you a quick example of how can you create a reliable SQL agent where user can just ask natural language question and agent can turn that question into a proper SQL query to get data from database and generate answer. And I think this is a great example because you can totally just build a free-form agent where it just have basic tools like getting a table, getting a schema and let it decide what are the order to execute. The problem if you ever try to build a SQL agent is that often they are not generate the right SQL query. So, we actually want a good retry mechanism in there. If you can break down the process into a more linear and controlled flow, we might get a much better result. So, we can actually use a graph to build a agentic system where it follow a very specific process. It will try to get the most relevant data table based on the user question first, and then it will also try to get a schema of those data table. And then it will try to generate query, reflect a bit and extract the proper query, and in the end to execute it. If the execution result is wrong, then it will send back to a query generation step to generate again. And if success, they can generate answer. So, I'm going to quickly take you through how can we build this. And team from LangGraph already compared these two type of agents. One is free form agent, another is those kind of multi-step breakdown agent process. You can see the simple evaluation score. The breakdown version answer score is around 0.67, while the free form agent is only 0.53. So, it is significant improvement compared with free form agent. And I'm going to quickly show you how can we build this? So, I'm going to open the notebook, and then we'll try to download a simple database to the local machine, install LangChain, and then use LangChain SQL database tool, and then quickly test if the SQL database is already running. You can see it returned this database. And you can also download free software like DB Browser to quickly take a look at any database to understand structure. And here, if we take a look, it loaded all the data table as well as reading schema properly. And next, we will use LangGraph to build a tool node. So, tool node basically is a special node or step in the graph that you can just give this graph a tool like execute SQL query, get a specific table schema, things like that. And for this one, we will set up some error handling so that it can send back error message and retry. And next, we will also test the two default tools from the LangChain SQL database toolkit. One is list all the table tool, another is get schema tool. And you can see below, it is working well. And we will also create a helper function called DB query tool. This is basically tool that agent can use to run the SQL query generated. And then, we will also quickly create a query check tool. They can basically review the initial query generated, fix any issues. If the initial one is wrong like this typo here, you can see that it will automatically self-correct to making sure the query is more likely to be accurate. And those are all the tools that we need. Next, we can start setting up the graph. So, we're importing a few different libraries. The state here will be pretty basic. It's just a chat history. Then, we will define a new graph with this state. And firstly, we want to add a node to get relevant table based on the user query. So, I will define a function called list table node, where we will compose a message about the user message, and we will just quickly force a tool run. So, in here, I will put a AI message try to run a specific tool, and then I will do the tool message by invoke the list table tool that I like. So, this should return the message about the user request AI get it, AI try to do the function call to get all the tables available, and then I will pass on this message to get agent continuously generate a request call to guess schema tool to guess schema of relevant tables. And in the end, I will return the full message. So, this function will basically get the relevant table, as well as generate message to try to call the guess schema. And then, we will add this node to be list table tools. And then, we will also add second node, which is guess schema tool. And after that, I want to create the query generating node. But before we do that, we will create a tool called submit final answer. So, this is tool that query generation node can call if they think they already get the final result. And then, we will give a prompt of query generation system. Quickly create a chain for the query generation, and then define a function called query generation node. And here, you can see we put some logics. That the large language model here only should have one tool, which is submit final answer. If the tool is not submit final answer, then we will put a error message here. And then, we will adding those node to the graph. This will basically look at the initial query generated by this query generation node, and do some fix with the function that we created earlier, which is query check. And in the end, we will add this node of SQL query. Uh and the last part is the part that is going to quite interesting. So, if you remember, there will be special type of agent that is conditional, that can allow you to control where these things should go next. And this should only have three potential output in the workflow, correct query or query generation. So, if the last message from the query generation is tool calls, which means it actually call this function submit final answer, that means the workflow has been finished. So, it will return end. But if the last message is error, then it should retry the query generation again. Otherwise, it can return correct query to continue the workflow. And that's pretty much it. Now, we can just connect everything together. Firstly, go to the list table tools, and then go to get schema tools, and then query generation. After query generation, we should run this should continue function to decide where to go next. And if it's correct query, then go to execution. And after execution, we'll go back to query generation to validate whether the answer is correct. And we can compile. And you can run this to quickly visualize the graph you just created. And if we try to quickly run this graph by asking which sales agent made the most in sales in 2009. You can see it returned the sales agent who made the most in sales 2009 is Steven Johnson with 164. And if we go back to the DB browser, you can see among all those table, for that question, we probably would need to look at employees. And if I go to browse data for the employee table, among all the sales support agent, Steven Johnson does look like made them. And this will require run the query across invoices well to calculate all the invoice relevant to this person. And we can also stream the result to see the step-by-step. So, first, it run this tool, successfully identify that it needs to get schema of these two table in employee and invoice. Then, try to get a schema. Then, it generated query for three times. Only third time it seemed to work, then did a correction, then try to reflect and review the query, executed. So, this example of how can you create a more reliable agentic workflow with graph. It definitely feels a lot lot more complicated than creating free-form agent. But, the trade-off here is that you will have a much more controlled behavior for those agents. So, I'm really keen to see what kind of interesting projects you're going to start building with those kind of graph flow engineer type of projects. If you enjoyed this video, please consider giving me a like and subscribe. I will continue posting interesting AI projects. Thank you, and I see you next time.