Building Knowledge Agents to Automate Document Workflows

Key Takeaways

Um, cool. Hi everyone. Uh, welcome on a Tuesday. Thanks for thanks for being here. Uh, I'm Jerry, co-founder, CEO of Llama Index. And today the talk is on building knowledge agents to help you automate different types of document workflows. Um, so I'll try to talk for about like two minutes and leave like five to eight minutes for questions. Um, and okay, let's get started. So, a big promise of AI agents is basically making knowledge workers more efficient. Um, if you've looked at any B2B SAS website these days, it's really about how can you use AI to help improve like knowledge worker productivity. If you think about the higher level like kind of uh like value metrics, it's basically lowering cost, saving time, and then also improving decision-m with more data. Um, but you know what does knowledgework automation actually mean, right? Is it just rag chat bots? Clearly, it's not just rag chat bots. Um, and so if not, what are the architectures that can actually help meaningfully automate different types of knowledge work? For us, we're really excited about the potential of unstructured data. Um, and specifically the potential of automating knowledge work over different types of unstructured data within documents. So if you think about like PDFs, PowerPoints, Word docs, um you know, a lot of humans in many different industries historically have needed to actually read a lot of information on a page, figure out what's actually going on within the document. you know, use it to basically synthesize insights and then also take different types of actions like create a new report, you know, uh use a computer uh enter some data into a new system or do other stuff on top of existing data for the first time. You know, unlike traditional computer like heruristic based programs, AI agents powered by Grenai can actually automate this entire end to end workflow. One of the biggest pieces of or one of the kind of most important pieces of an LLM is the fact that it can inhale a lot of unstructured text tokens um and then you know actually automatically use that to basically uh take actions make decisions instead of you having to train a very task specific model on it. But to make this work really well they need access to the right context and architecture. And so today we'll talk about two types of agents to really automate knowledge work. Um there's assist of agents which are kind of your standard chatbased interfaces whether it's chatgbt or deep research. Um and the idea is that the human is like interacting with uh the agent to uh with a high degree of human in the loop to basically get the information that the human wants faster. Um the other type of agent is really this idea of an automation agent which runs maybe a little bit more in the background can uh run a multi-step process and typically can automate a routine task. Um so it can take actions on behalf of the user maybe requires you to interject a little bit less but it'll solve like one core workflow and solve it really well. To make either of these agents work you basically need uh two things right? If you think about any agent architecture in the general sense, um you need really really good tools whether connecting through MCP 80A or just you know through function calling um to basically surface relevant context and provide the right external interfaces to let the agent take actions. And then you also of course need the right agent architecture um something that's very general and can do a lot of actions or something that's a little bit more constrained. And it's really a way of encoding your business workflow as well as the tasks you want to solve into this agent architecture. Um, a a lot of knowledge work is rooted in documents. Um, if you think about kind of the the columns on the left are basically different classes of knowledge workers and obviously it's not comprehensive. This is just a kind of initial list that we came up with. And then um the columns uh in the in the middle and right basically outline the different types of agents that can help these types of knowledge workers. So you know for financial analysts um they might want to perform due diligence on private or public reports, generate investment memos, um do different types of things. Um you can also uh they do a lot of manual data entry from existing documents into for instance Microsoft Excel. A lot of these things can be uh handled by AI agents either in a research or co-pilot setting or in an automation setting for administrative or back office operations. Um you can have like you know assistance over HR onboarding uh legal assistance um or you could have these like automation agents actually run business processes end to end like invoice uh processing and reconciliation reviewing a contract uh claims processing for insurance um and and you know many many more tasks. And so, as you can tell, a lot of knowledge work really depends on you being able to process a lot of information within a PDF or other file formats, being able to transform that data in some way. Um, and it basically boils down to how can AI agents actually help handle a lot of these tasks end to end. Um, for those of you who are not familiar with LM index or for those of you who are familiar but know of us as a rag framework, uh, you know, we started in the early days of tragedy. Um we started as an open source framework um and basically built a lot of tooling uh and orchestration layers to help you build LM maps over your data. Nowadays we've become a comprehensive platform to help you automate document workflows with AI agents. And so there's really two main components to Llama Index. Um there's the like Llama cloud layer which we'll talk about in just a bit. It's basically the core document intelligence layer that allows you to process and structure your data in the right ways and provide those as tools to an AI agent. Um we have an agent framework uh available in Python and Typescript which contains an flexible orchestration layer tools and integrations with hundreds of different providers as well as application templates to help you build different use cases. Um and a lot of the use cases that we've started to focus on really are these documen ccentric use cases that again help a business uh user get more information faster or help automate um you know increasing chunks of their day-to-day workflows. And so the overall structure of this talk will be the following. Um the first piece is really building a document toolbox. Um providing the right set of tools that an agent can have access to to basically operate over your docs. The second piece is the architecture like different types of agent design patterns. Um there's two types we'll focus on assistant to automation based UX's and then the last is putting everything together into uh comprehensive document agent-based use cases. So first let's talk about this idea of building a document toolbox. Um how many of you are familiar with rag? All right, I think it's probably most of you. Um I think the idea that I want to impress upon you guys is basically this idea that you know rag is kind of a narrow subset of how agents can actually interact with their data. Really what you want to do is provide the right external service uh uh interfaces and services to the AI agent. So it has a rich set of tools to both read understand different types of enterprise context and also manipulate and take actions over it. And I call this like a document toolbox. And to actually create this document toolbox, you need the right pre-processing layers from your unstructured data sources to actually process and structure your documents in the right format. So some examples of what you might want to do, and you might be familiar with this from a rag setting, but being able to connect to a data source. So setting up a data pipeline to SharePoint, Google Drive, S3, Confluence, um, as well as any other data source. um making sure this pipeline supports you know incremental data updates and scheduled syncs um so that you can periodically refresh the context available in your downstream storage systems. The next pieces include you know parsing and extraction be able to actually process any document type uh even if it's very very complicated and be able to actually encode that information or translate that information into text tokens that LM can understand. Um, this includes tables, charts, scans, um, and and just a lot of documents have a lot of really weird formats. You might want to have an extraction layer, um, to basically transform unstructured data into structured formats. You can then take the structured data and either put it into um, a metadata as part of your chunks that you put into your vector database, or you could take this metadata and put it into a downstream uh, structured data warehouse. And of course there's some aspect of indexing and a rag setting. This includes chunking, embedding, indexing, advanced retrieval. A lot of these are basically just a set of operations that you probably want to do on top of your data to basically make it ready for an AI agent to interact with. And once uh you know making it ready basically means creating the right set of tools over it. Um, and so this could include, I'll go into this in the next slide, a vector retrieval tool, which is basically just the rag piece with uh, you know, metadata filters. It could include a file tool that lets you, you know, understand uh, different properties of different files, retrieve entire file contents, um, as well as, you know, be able to really traverse, uh, the the different types of file directories that you have. Um, a SQL tool to let you get aggregate analytics and insights over the metadata. um as well as manipulation tools. So that's basically what I talk about uh in this slide. So you know if you have this pre-processing layer um some of the examples of the tools that you might want to generate from this pre-processing includes a file lookup uh retrieval structured quering um and manipulation. And the the the kind of powerful aspect of this type of framing is that you know by creating this type of uh the set of tools and standardizing it according to a protocol like MCP or 80A this basically means you can plug the set of tools into any sort of AI agent front end right whether it's your own handbuilt handcrafted agent or into claude tragic t cursor uh your Microsoft co-pilot your favorite AI tool out there um just to contrast this with you know naive rag um naive rag is basically you have uh you know I'm sure many of you are familiar with this but you know you you basically have a query um you directly feed it to a vector database or you embed that query um feed it to a vector database get back the text chunks and then use the context to synthesize an answer. Um the more abstract way of looking at this is some agent layer interacting with a tool layer. Um so the agent um you know could be arbitrarily sophisticated or unsophisticated. It could be a react loop, a very general agent reasoning loop. Um or it could be super constrained and one shot. And then the tool layer um you're basically providing like a retrieval tool with a set of parameters. You know it takes in an input query as well as metadata filters. But then you can have other tools too. For instance like a file tool as well as other surrounding tools like web search uh tools to connect to other types of enterprise context. So the agent actually has a flexibility to incorporate the tools as it see fits uh as it sees fit to actually uh solve the task at hand. Right? And so some of the um expanded capabilities uh that the tool interface provides in contrast to naive rag is the agent can choose to naively or sorry uh can choose to smartly or intelligently uh retrieve entire documents um retrieve like screenshots of pages get links to other documents um all by providing the right output and context to the AI agent. Um, another tool is obviously like text to SQL. Um, probably pretty relevant in the data brick setting. Um, if you actually process even unstructured documents and you have the right extraction and structuring layer, you can create structured data from those unstructured documents, create a CSV uh, or even, you know, dump that data into a structured data warehouse and then you can basically interact with those types of interfaces through relatively standard methods, right? there's like text to SQL, text to CSV methods and a set of challenges required to actually uh do that at scale. Um, one kind of subtopic that we did uh we've talked about for the past year or two, but we do find to be, you know, a pretty prevalent issue in the enterprise setting is this idea of complex documents. Um, a lot of documents can be classified as complex. Uh, they contain embedded tables, charts, images. Um, they have headers and footers. They can be multilingual. Um, they can have like scanned images interled with actual text. And one of the core issues that anyone faces when building like a naive agent is that if you use an off-the-shelf document processing solution like PI PDF, then it's going to mess up the text. And then if you feed up messed up text tokens to the LLM, it's not really going to understand uh what's going on with the context. And so one quick note here is basically um we've invested a lot of effort in uh in targeting this idea of VLM native like parsing and extraction. We were probably one of the first folks to really understand the fact that instead of like traditional OCR methods, you can actually leverage both LLMs as well as VLMs like the you know being able to screenshot a page and feed it into one of these models um for high quality parsing and extraction. Um so you know in contrast to traditional ML methods which require like task specific ML models trained on domain specific data um and they're not typically robust to layout variations other document formats um we basically combined genai models with best-in-class paristic parsing techniques to really get the best of both worlds. Um the baseline obviously is you just screenshot the page and then you feed it into claude. This works okay for like probably 80% of documents. Um but Claude gets really lazy at the longtail. Um starts hallucinating uh trailing off doesn't actually give you the full text output. And so when you combine it with both parsing techniques and then add an agentic loop um to basically add more runtime test time compute on reasoning tokens, then you get really really nice performance. Uh you get higher general accuracy than traditional approaches. Um and then you know of course there's still a long tale of issues to work through but we're basically adapting a lot of the latest models out there like chat GBT quad gemini and tuning making it agentic for really really good highquality document parsing and extraction and it's really the first step towards making sure your data is right. Um this is a benchmark that we ran across uh pretty much a bunch of other tools out there both public and also private in terms of document parsing. Um we have a ton of modes within llama parse our document parsing service. Um it incorporates a lot of the latest models. So you know we tune Claude Chadi or OpenAI 4.1 and Gemini 2.5 Pro make it agentic um and basically measure um according to a few different eval metrics alum as a judge as well as edit distance and when compared to other open source or proprietary solutions we typically do uh much better. Um, and so Llama Cloud really is that layer um, that document toolbox to help you pre-process and structure your data and the right format for your AI agents. Um, and contains uh, the document parsing stuff that you just saw in the last slide. It also contains structured extraction. Um, so besides just turning your documents into markdown, uh, we'll also turn your documents into a structured output format. Um, and then you know the idea is to again process and structure your data to expose a set of tools. And you know, we're working on a feature to make it natively MCP compatible. Um, but some of the tools include retrieval, extraction, manipulation, and more. Um, just to run through some features really quickly. These are just subsets of the overall capabilities. Um, one feature is, you know, aic parsing with layout mode. Um, so we'll, you know, draw the bounding boxes on the page, identify the different elements, and then dynamically figure out how to parse each se each section in the right format. So if it's all text, you don't really need something super complicated. If it's images or charts, um you use VLMs under the hood. This also gives you back visual citations back to the source document. So instead of just linking back to the sentence, you actually get the bounding box of the element. Um another piece is multimodal extraction with reasoning. Um we have reasoning and citations on the extraction endpoint too. So if you get back structured outputs on every single key and value, you can get back citations and reasoning back to the source text. And then lastly, um you know, we also have an indexing retrieval feature. And the idea is, you know, like we actually have, um some kind of cool capabilities under the hood to uh dynamically return the relevant set of context instead of just doing fixed top search. Um there's a composite retriever which will route your query to the right data source. Um have some sort of auto routing to determine whether to return chunks or documents. And the nice thing for you is that if you use a service, you basically get back a unified interface um like a single tool they can plug into an AI agent. And uh one demo that we just released last week and this is something that I haven't really seen a lot of people talk about. There's, you know, obviously other players in document parsing is Excel capabilities. Um and so a lot of knowledge work happens in Microsoft Excel. Um traditionally it's been unsolved by LLMs. Um because Excel sheets are typically visually structured for humans. If you look at even this like PNL sheet right here, it's not really a 2D table. Um there's a ton of gaps uh in the rows and columns. So if you run standard text to CSV techniques over it, it's not going to do very well. Um a vector retrieval, obviously don't do rag over an Excel sheet. You're not going to get back good results. Um the best baseline, honestly, is just having uh the alm write code um to try to understand the file. But even that is oftentimes not enough. Um so we basically built um you know an Excel agent that is capable of both uh two types of tasks right it's able it has the right tools right again it's not doing rag but it's able to do manipulation so data transformation over the Excel sheet to normalize structure the data in the right format and then also because it has the right set of specialized tools do agentic QA so it can answer questions over cells in any of these sheets. Um, in this example, we're basically uploading that Acme P&L sheet. Um, and then this example right here, I know it's a little faint, is just showing you how you can structure that Excel file into a normalized CSV. And then once you get to this Q&A assistant, you can upload that same Excel file. Um, and then ask different types of questions like I I know you can't really see the screen, but what are the top three expense categories? And then it uses specialized tools under the hood to try to answer the question at hand. So this is available in early preview. Um if you have Excel use cases that are of this nature um you know you have these unnormalized Excel spreadsheets that need some layer of understanding to really help your business users accelerate their workflows come talk to us. Um as I mentioned the best baseline like rag sucks don't do rag over excel the best baseline is just code interpreter. Um so we benchmarked against openi code interpreter with 4.1 and 4 40. Um you know that gets us like 75% accuracy over a data set over financial data. Um we with a set of specialized tools you can really increase agent performance by a lot. Over here you know we actually got accuracy up to 95%. Um and then the human baseline is is actually 90%. Um a little bit of a snippet on how this works. Um for Excel specifically we we did a lot of fancy stuff under the hood. um we did some sort of like RL based like structure understanding where depending on the structure of the sheet we actually learned a semantic map um and a data representation over various types of like you know unnormalized tables you might have once you actually create the right data model you can then expose those as tools to a specialized agent. Um, so again, the baseline is just the LLM is writing code using some commonly importable library in Python or TypeScript. Um, and basically building a lot of this stuff from scratch. And you can think of this as really just accelerating the time to value and also giving you a higher degree of accuracy. Great. So that was the document toolbox section. um you know basically just wanted to talk about the fact that you really do need this data layer and the right set of tools to really interact with document context and this provides the foundation for helping you automate knowledge work. The next piece is now you know you want to to define the right agent um according to the use case and there's really two types of agents. There's assistant UXs as well as automation UXs. Um, as many of you might know from building agents yourselves, agent orchestration ranges from something that's a little bit more constrained. Um, you know, you explicitly define the control flow as a human um to unconstrained where you really let the agent handle the control flow and reasoning and dynamically select the right set of tools to uh to solve the task at hand. And the assistant UX kind of falls a little bit more on the unconstrained side of that spectrum. Um, and you know, you've all used these types of interfaces. If you use chatbt or claude, uh, or you know, I guess the cursor agent with tools, um, it's basically a human interacting with some sort of like relatively unconstrained agent at every step of the way. Um, the UX is typically a little bit more chat oriented. um you know I think we're all familiar with that type of interface at this point where you as a human need to supply the natural language or file and then the agent will basically dynamically figure out what tools to use and then return a response and then you basically interact with it in back and forth sessions to produce some sort of output. So this type of architecture there's typically a few like a few more tools with a high degree of human in the loop um and it's usually a little bit more unconstrained. The goal of an assistant UX from like a business user perspective usually is I mean it saves a little bit of time because you don't have to go and dig up the information. Um but it's also just like helping the user get information a little bit faster in a true like co-pilot like setting. It's really helping the human, you know, get more stuff faster so they can get more things done. Um I mean this is just a subset of the use cases. There's enterprise search, you know, financial diligence, uh, customer support. Obviously, chatbt is is an assistant UX and it can basically do a lot of different things. Um, but I want to contrast that to, uh, what I call like automation UXs. Um, automation UX's uh, are those we're seeing a little bit more of these days as people are building more multi-step agents. And they're basically end-to-end workflows that require a little bit less human in the loop and can help process routine tasks and and um you know typical business workflows. And so it's taking an existing job that's typically done by a human but done in a relatively repetitive manner um and then just like codifying that into an agent workflow. You don't necessarily need to chat with it at every step of the way. you probably do need human review before it goes and makes uh you know a final decision at the end. Um but it's also a little bit more constrained, right? It does one type of task and does it relatively well. Um and so the goal of this really is to help uh just completely solve some existing workflow end to end. doesn't necessarily help the human increase like the amount of data that they have for like general decision- making um but in general increases like operational efficiency. So some example use cases here right um include basically any type of work any type of knowledge work where you're manually reading data and try to put it into a computer system. Um this includes you know invoice processing and reconciliation. This includes like contract review. Um this in this includes like you know processing insurance claims, data sheet extraction for deeply technical specs and more. Um this video right here is just you know a community example we built uh with uh invoice processing. Um this this is available as a completely open source repo um on our GitHub. All right, go to the next section. So, um I think as we have more and more agents and more agents are going to fall in one of these two categories generally or maybe a little bit of a spectrum between the two um you can start thinking of uh this this is a bit more of a vague less fleshed out idea of an agent backend and front end. So the agent backend consists of like automation agents which either run in the background or could be triggered through some sort of API call and you know in in the form of like uh automating knowledge work. They basically automate data ETL structuring but also decision-m um and they basically just run in the background and tackle a bunch of data in a batch setting. Um they can also process and structure the data and provide uh and expose that through a set of tools. And there's more assistantbased agents which are a little bit more userfacing. Um, and humans actually interact with it. And you know there it's basically the front end uh for humans to interact with. Thinking about it in the context of like MCP, you know, assistant agents are a little bit more like the MCP client. Um, these automation agents typically run in the background and maybe are a little bit more of like the MCP tool. Um to that point, you know, uh one idea is basically having these automation agents encode certain types of document workflows and have them um act as MCP servers. So the powerful idea here is imagine you built some sort of AI agent that was relatively constrained and solve one specific type of task. Well, um, in this video example, it's basically, um, you know, being able to actually extract out every single table of financials from, uh, you know, a PDF of Fidelity's like different funds. Um, and so each like that document contains a list of different funds. Each fund has financials attached to it. And then, you know, you have this workflow that actually just executes this task. It has a defined schema. it knows what type of output uh what type of financial metrics it's trying to extract. It consolidates all the information into a structured CSV and returns it back to some sort of MCP client. Um so in this example, we basically define that as a tool and hooked it up to Claude and Claude is able to then you know take the outputs of that tool and then automatically write its own code to basically generate a final report at the end. Um so if I go right here you can basically scroll through um these are the output examples. Um you know the the advantage of MCP tools um part of it is to really integrate external sources of context. But I think another really interesting thing especially in the context of automation agents is to really encode these types of business processes have those as workflow tools and basically standardize them across business users. So as a user, as an end user, you don't have to do the work of actually prompting the entire thing yourself. You have a set of standard tools that makes it really easy for you to use an AI agent to achieve the task at hand. Now lastly, to basically put it all together, um you know, here's some real world examples of just uh document agent use cases. Um so as I expand this a little bit. Okay so these are basically just like real world use cases of both assistant and automation UX's um financial due diligence is something we talk about a lot. Uh one of our customers is Carl and uh you know one of the core use cases is a this idea of building like an end-to-end leverage bio agent. Um you have a data room filled with private and public financial reports. you need some sort of process to really process and structure all the financials and data from a list of heterogeneous document formats from PDFs to Excel sheets um structure them in the right way. So that's what you know an automation agent typically does um and then create some sort of uh tool interface effectively for an assistant agent to interact with. And so the end result is that it can be a complete end-to-end process where a human is in the loop to actually review the extractor results uh from all these different documents, make sure at least the numbers are correct and then feed it into some downstream process that constructs a financial model or creates a chatbot that a human can interact with. And some of the core benefits is really just um you know being able to speed up the operational work that typically takes a team of analysts like a week to do. um and also add increased decision support because with this natural language interface you can actually get more insights over the data than if you actually try to manually review it with your own eyes. Um another one of our uh fun customer examples is basically seam and this example is very much in the assistant UX category. I think a lot of you guys are probably building similar things in terms of enterprise search chat bots. Um it's basically the generalization of rag or like aentic interfaces that help you understand, interact and do research over your data in different ways. Um and so you know you can connect to a variety of different data sources. It could be a Microsoft SharePoint, it could be an S3, it could be Confluence. Um and you basically build targeted chatbots that have full enterprise context over that piece of information. You add a system prompt and you create a guided user experience so that a specific persona within the enterprise can get the information that they want to help them uh solve uh you know whatever task they they have at hand. Um another fun example that's almost purely in the automation space is this idea of like technical data sheet injection. So imagine if you're an electronics semiconductor or even like construction or uh manufacturing company, you're probably dealing with like documents that have really heavy and rigorous technical specifications and oftentimes it's super non-standardized and there's like a PDF with like a thousand pages in it and you really need to understand these diagrams and you know translate it into some sort of structured format. This is this example right here. Um, we're working with an electronics company that employs an entire team of technical writers to actually lift out the information present within these documents and translate them into structured data and can take them uh, you know, weeks of manual effort to process uh, some collection of these documents. So, luckily with uh, an automation agent interface, you can have AI agents basically do the data backfilling and structuring uh, automatically. Of course, it's not completely 100% accurate, but with especially with some of our stuff, you can get like 99 point something percent of the way there and then flag and include aspects of like uncertainty quantification so that even in that like last mile where um you know the agent thought it might have gotten it wrong, um it at least flags it right so the human can review and correct the results. Great. That's basically it for the for the talk. Um, for those of you who are, um, this is just a general overview of Lom Index. You know, we're an accurate, customizable platform for helping you automate your document workflows with AI agents. Uh, we're backed by Greylock Norwest. Uh, we have a very popular agent framework um, that's used by 300 plus of the Forge 500. Um, a few million monthly downloads, 5 million plus. Um, plus, you know, 150 to 200k plus signups on BAM cloud. So, thank you all and I think we have a little bit of time for questions.