LlamaParse vs  LLMs: Live OCR Battleground

Key Takeaways

[music] [music] [music] [music] [music] >> Hello everyone and thank you for joining. My name is Murtaza and I lead marketing at LlamaIndex and thank you so much for joining our session on Llama Parse versus LLMs a live OCR battleground. Um today we're joined by hundreds of folks joining in on the call. Um as a couple of reminders and housekeeping rules, the session will be recorded. You can feel free to enter comments and questions in the comment section and you can we'll get to them at the end of the session during our Q&A. Um for folks um who are joining in remotely from different parts of the world, feel free to share in the chat where you're joining from, what you're looking most forward to and as always as you have questions, feel free feel please feel free to ask. Without further ado, I wanted to invite our chief guest here, George. Um George is our head of engineering at LlamaIndex. He is a Harvard trained lawyer and if that wasn't enough, he is one of the best engineers I know. Um he leads our engineering efforts at LlamaIndex and he put a ton of work into this presentation and so I hope you all enjoy it and George, over to you. All right. Thanks Murtaza. Hello folks. Um nice to meet you all. Uh Hello on this uh early Thursday morning. Um today we'll be going through uh a few decks and a few real world examples on how Llama Parse performs versus LLMs. Um we'll go through some history. Uh let's see. Um beginning with a problem in field introduction of why you should care about document processing, what is going on when you're dumping documents into LLMs and kind of how the field got to this point. >> [snorts] >> Uh we'll then be going into a technical breakdown of the solutions that have evolved over time. Um we'll go from, you know, early 1980s all the way to now. Won't take too long. Um then we'll go look at some of the failure modes that we have seen in the wild when implementing LLM adjacent or agentic processes uh in order specifically to extract data from documents. Lastly, we'll go through the live battlegrounds. We'll have a chance to go through some real documents as well as some of the documents that have really perplexed us as a team um as we've gotten in. And towards the end we'll we'll save some time for a live Q&A. Um Let's get going on. So, to introduce the problem that we'll be talking about today, um we will be focusing on document processing as a topic. So, document processing as a field has been fairly straightforward, but the last 10% of documents that people usually see usually break their systems. Um the majority of the data that we see from our enterprise customers, the majority of the data that is in enterprise is usually stuck in complex document format. Now, this can be scanned, this can be OCR'd, this can be directly in text, but what we've found is that when data is stored into underlying formats for human viewing, which is usually in PDF, uh machine extraction becomes fairly difficult. Um and that's due to a combination of the underlying technology used to store the documents itself. Um so what I mean there is, you know, data that's encoded and that seems straightforward in a PDF and this also applies to Word as well as other formats, uh can be represented as glyphs rather than characters. Um what seems to be a straightforward reading of text to you can be a set of absolutely positioned values in the PDF. So, it's very difficult for uh vision models as well as normal uh language models to decipher what's going on. Um you get to see a combination of why document processing is hard when you begin looking at real world reports. We see a good number of financial reports, schematics, um manufacturing specifically, as well as kind of insurance tables that become very complex in our day-to-day that begin breaking down the barriers of what you think a sane OCR process would handle. Um we have a lot of complex financial charts you can see over here um that has, you know, data embedded inside of it and without complex extraction capabilities, handling charts, tables, graphs, etc., uh you're not going to get the results that you want out of your system. Um usually if you use the traditional OCR here, you would not be able to pull the table out. And I'm going to just uh switch tabs real quick to show what I mean. So, here I have one of the um I guess this is earnings report from Uber pulled out. Um you can see that there is some complex data here, you know, you have a few diagrams, charts, etc. Um and when you parse it, you're really looking for a textual representation um of what the underlying data is. So, you can see here, you know, there is an ability for OCR uh OCR combined with LLMs in an agentic process to pull this type of data out. Um you can decompose complex tables and complex charts in a much more comprehensible way. >> [snorts] >> Now, jumping to charts, tables, and schematics specifically, um you might be thinking, "Isn't this already a solved space for PDFs, for other data types?" Um the answer is actually no because you can scan documents. Uh there is no requirement by, you know, the PDF format to have tables specifically in a certain way. There is a PDF spec that technically defines how structured layout should work, uh but the real world scenarios that we see, you know, when you convert from docx to PDF, when you convert from pretty much any proprietary format to PDF, each developer has implemented their conversion mechanism their own way. So, when you see a table visually, it might look the same, but under the hood, everything is completely different. Everything is composed separately. In terms of how these grids are shown, um you can see an example to the right here of kind of just encoded characters as well as how lines are drawn in order to represent a grid. There's no semantic connection between the lines and the text themselves. So, if you think about what this would look like to an agent, this would be equivalent to a bunch of absolutely positioned characters on a HTML screen. Uh things may be out of order, things may completely not make any sense uh if you read it without rendering the whole thing. So, it's not straightforward. Um let's see. I think we might have lost a slide here, but um in addition to this, uh the there is complexity in how a parser understands the uh intersection of what are the boundaries of a table and how a table actually uh should be rendered in markdown or HTML if you were to try to render it. Um and that's because if you use a raw LLM provider um or just dump your document into an LLM, it may be able to decompose different components, but it will not be able to isolate tables correctly. So, if you have two tables next to each other, they can end up being merged. Um and so there there needs to be some kind of rule or harness that's given to your agent as it's going through this translation process for the PDF. Um a chart similarly has the same set of problems. Um bar charts, pie charts, etc., there is no format that's defined in the PDF spec to support this or if even if there is, uh it doesn't define it fully in a way where people actually follow it. Um there are no real ways to render a chart in a way that is easily read back by an LLM or a LVM today. So, we'll go from there to thinking about how this worked before uh there was the magic of transformers. Um in the old school IDP process, so intelligent document processing, this is kind of in the world before transformer models were popular, before you could just throw something at an LLM and say, "Please do something with it." Um the old world worked in a set of predefined pipelines. We can just call this the pipeline world. Um and in this world, you would break down the document OCR process into multiple steps that would require human computer curation along the way. So, you would find this cyclical improvement cycle that a lot of companies, contract management companies, or just traditional scanning companies would implement uh where you would begin classifying documents into specific types that you would support ahead of time. You would maybe localize what data you wanted extracted and then you would perform kind of old school OCR um in, you know, text recognition, alignment, uh skew adjustment uh before going into your extraction validation process. And there's a lot of caveats here with how you can actually uh determine how confident you are in extraction as well as your reasoning in extraction. Um in the old school IDP world, I would say it was not very possible to uh essentially one-shot a document because of complex document structure being in the fray. Um so we'll go from there into understanding how the rest of the system was built up. Um and to frame this, I kind of think about it in three different stages. Um there is this old pipeline era, right? And this this applies to a lot of developers that I've seen, like Tesseract or, you know, even more recent, this would be convolutional or recurrent neural network architecture where you might have to fine-tune or adjust on a specific data set. You might have to fine-tune on your language. Uh and the hard part here is that this rigid process makes it very difficult to handle unstructured documents. These This pipeline approach worked very well in handling structured documents because, you know, all of these have very specialized models um as long as your documents are similar in nature. There's usually no issue. Um and that's where kind of we've evolved to the model era. So, the model era is kind of, you know, once transformers were a bit more robust after the 2020s. We're in the 2020s. Time's moving fast. Um people became used to dumping text and eventually screenshots or raw documents directly into the LLMs. Um vision-based models became popular, you know, GPT-4 came out with vision capabilities, Gemini, Claude all came out with vision capabilities kind of end of 2023. Um and a lot of people kind of just put a pin in it and said, "Great, I have solved document OCR. We don't need to look at this again." Um and what we found is that just using the vision-based capabilities is not sufficient. Uh you need a very rigid harness around it. Um and even better, uh you know, giving a hybrid approach, uh which is what I'll talk about next, uh usually gives you much better capabilities. Um what we found is, you know, vision-based models can't handle high high density, high um granularity content very well and relying purely on a vision-based model without the proper harness usually leads you into a few um I'd say failure modes for LLMs. Um so, the right approach here that we'll be talking about um in the end is going to be the hybrid approach. Um so, this approach is going to be a combination of using some traditional layout detection, uh utilization of kind of traditional OCR, and combining that with the underlying data, uh of uh you know, the raw document or screenshots of the document, passing that to the VLM, and then creating a iterative harness for you to be able to extract data, and even if you do fail, step back and resume from a certain position without going into those failure modes. Um you know, VLMs handle visually complex regions pretty well if you isolate it um to the VLM. Um you know, complex, misaligned, non-standard, unlabeled tables, unlabeled charts, um these are all very difficult to handle in the traditional pipeline approach because there is no trained model that handles this on the convolutional side and extracts data very well. But vision-based models are able to reason through uh these hurdles fairly easily. So, by combining, you know, rigid structure with the ability to zone in on areas of high ambiguity, we at least speed up the process and as a second, uh bring more structure in making sure the harness and the VLM does not run off uh the rails. All right. Um and kind of continuing on on why you would actually care about each of these different spaces, um there is a solution space here in terms of quality, latency, cost, and a few other features that you might care about as a developer that you that people should be thinking about. Um if you have not thought about it, um I'll go through some of the high-level ones here. Um so, in your traditional OCR++ or advanced IDP process, this is kind of stuff that you may be able to run locally at this point. Um latency is usually very fast, cost is very low. You might want to use this for real-time chat apps. We recently just released uh an open-source offering called Light Parse, which is meant to kind of address the space where um if you want something that is available for your agent to just instantly understand what's going on to a document locally without spending more than a couple of seconds uh processing it, you can now. Um this obviously existed before, um but there is an LLM output aspect of positional awareness, layout awareness, and table grounding that is missing in most PDF parsers and has historically not been benchmarked very well. Um I'll talk a bit more about that down kind of down the road. Um the other thing that I want to highlight is with LLM native or kind of multimodal capabilities, um there is very high, you know, I would say it's high high flexibility. Uh your goal when you use a LLM native or multimodal approach is usually to one-shot your documents and not have to think about this entire IDP process. Um and that makes it very appetizing for developers as well as companies with complex documents. Um but there is a failure mode in that LLMs usually do not provide traceability or grounding. Um so, you know, LLMs are very prone to hallucination. You need to keep a very close eye on making sure how your LLM grounds its output relative to the input documents. Um if you give it the Declaration of Independence and swap out half of the words, is it going to override your swaps? Um things like that can be issues with uh native LLMs. Um and looking into the last piece, which I think is relevant for most people on on cost combined with latency, um LLM native approaches usually have a tendency to go extremely high on certain types of documents. Um whether your thinking is low or high, it's likely that the model can enter thinking loops, um decide to begin implementing custom code to parse PDFs, which we have seen, um and your goal is usually not to enter those types of loops, uh and instead to have low latency with high quality without the model burning through all your tokens. In an agentic multimodal workflow, um or agentic multimodal workflow, we're able to control these costs a bit better, um as well as short-circuit even needing an LLM in certain cases. Uh you know, but even though we have used VLM native models as well as agentic models in the loop, there's still quite a bit of barrier with model hallucination as well as understanding the model thinking process to get to better outcomes. Um here uh I have some screens pulled up of hallucinated document outputs uh from specific models. Now, hallucinated may be a bit harsh because in some cases the tables or charts are printed or displayed in a way where you have to assume certain types of reading order or your when you read it as a person, your mind is automatically filling in the blank for certain pieces of data. It's In some cases unfair to penalize models for rendering tables or charts in a in the way that you would interpret them uh versus directly in the markdown. Uh but that's also kind of up to what your end use case is. Are you trying to give this to an LLM for understanding or are you trying to verbatim copy this document over? Um that's also why benchmarks are a bit uh skewed, I think, these days. Um on the right, top right, you can see uh um something that I'll get back to later in this presentation, which is on thinking modes as well as the quality, time, and cost related to each model. Um what we have found in our benchmarking is that for pure document translation tasks, if you allow the model to think more, um it will correct itself into uh worse worse results. Um and some of the reasoning there is because in the initial shot, you're most closely grounded to the original document. Um and as you iterate further with additional layers of thinking, the models are beginning to correct human errors, uh the models are beginning to inject their own preferences in for how the output should be formatted, um and you're moving further and further away from the goalpost. Uh we'll use this later um in our benchmarking comparison. Uh we're going to try to use lower zero thinking where possible to make sure we're not unfairly penalizing the LLMs. Um here's a few examples on uh you know, how complex tables can get as well as where models can fail out. Um you can have tables where the axes and charts are in completely different dimensions. You can have tables and charts where uh even as a human, if you had me to try to transcribe this chart, I would be spending a fair amount of time trying to figure out what is what. Um and, you know, many cases, the models are able to cut it very quickly. Um in other cases, the models will completely break down when trying to create these charts or uh textual representations of uh graphs. Okay. Um so, let's take a step back. Uh we've talked a bit about the history of LLMs and how you would build one or how they have been attempted to be built. Um we've talked a bit about the pros and cons of using an LLM or a hybrid harness approach. Um now, I want to talk about something that's more important, which is why you would want to build this um on the product side. Um the use cases that we have seen for why people care about OCR again today are really centered around better knowledge management and better uh orchestration of existing pipelines that they have. So, usually this means that people have begun caring about a few additional use cases, such as confidence scores and quality indicators of their extraction. Um people are fine with 95% accuracy as long as they can have humans correct the last few percent points if they really cared about accuracy. So, confidence scores are very important. Human in the loop workflows, quality control, automation that self-corrects. These are all kind of things that we take for granted and don't have a lot of visibility into when we ship our documents off to a standard IDP processor. We just get the results and hope they're correct. Bounding boxes and source citations are also becoming more important. So, this is the fact of you know, if I ask my LLM where did you even figure out this piece of information? There's very little grounding. Bounding boxes are missing. There's no source citation. At least it's not persisted properly in metadata when all you get back is the pure markdown. So, thinking about how we can begin persisting this is important. Structured layout and reasoning information is very similar to the previous points on structured bounding boxes. But, in this case we want a bit more representation as far as you know, not just text. I need to understand the hierarchy. I need to understand where tables are positioned, where charts are positioned. And even as a human trying to understand when you should be reading a table relative to a chunk of text is important because it provides context and positioning information that informs how you read the document. In a lot of OCR or traditional LLM approaches, reading order becomes messed up. You know, if you have a two or three column newspaper, are you reading the middle column first? Are you reading the higher up column first? Or are you not even reading it because there's an image in one of the columns? In latency and trade-off, this is a bit more obvious but but with LLMs and you know, non or sequential token models available, offline versus online requirements are a bit more stretched out. Before the worst thing you could do was kind of have a localized GPU intensive process. Now it's you know, a bit more of a cloud network latency intensive process to perform LLM based extraction. On building the right system for LLMs, I'm going to just compare our open source offering to uh PyPDF which is another document processor. I'm going to see if I can zoom in for you guys here. I don't think I can. I'm going to just zoom in directly using the screen. Apologies if you can't see this too well. You can see that in raw open source comparison of like PyParsers PyPDF where transformer models are not utilized at all, we're really focusing on what the intention of parsing actually is. Positional information is really lost in a lot of more old school processors. You would not be able to decompose this very well as a LLM model due to spacing restrictions. The model has a much poorer understanding of how layout is performed because of the lack of positional encoding. What we really try to focus on at LlamaIndex with kind of LlamaParse as well as LightParse is maintaining positional information so that the LLMs can decompose, recompose the actual document as close to being faithful as possible. And this is stuff that a lot of benchmarks don't catch or even penalize us on. So, there's a deep understanding that's kind of missing in the field for why you would care about certain types of layout, certain types of chart representations to be shown in a tabular format or with specific mark markings in HTML instead of markdown that is missing in the industry. And this is also what we'll be kind of diving into later today with the one-on-one battlegrounds. Okay, let's go back to slideshow mode. Okay, [snorts] so that's a bit about technical breakdown on what you actually should be thinking about when you are building out these parsing systems. This may not apply to you, but it's good to think about. >> [sighs] >> Now, thinking about this one-shot structure that developers kind of dream about which is I don't want to think about training my OCR model anymore. I don't want to think about um you know, if you did not pick up this specific chart, why you did not pick it up and how do I fine-tune my trained table or chart extraction mechanism to better handle this. Um we think about breaking structure down still as a pipeline. I don't think we're quite at the stage of agentic orchestration, however you want to call it, where you truly can just look away. And that's why we have the harness system in place. Complex data including you know, embedded charts, images, irregular layouts, footing mixed languages. We've seen all of this. If you don't prompt the agents correctly or the agents are not configured to analyze charts and tables with additional attention, it's going to fail out. You can see very clearly here in the equity research report, you know, it's you see a tons of these consulting reports pushed through the industry that there is a combined table and chart meant to inform how you interpret the highlighted information. And without it, you're going to be missing context when you perform a rag or agentic based search. You're going to be missing this when you query your data set and try to figure out what the information was that you very clearly remember reading, right? Um and so we have taken a look at this problem, you know, there's a combination of structured extraction that's missing, but also a combination of decomposing the metadata so that it can be easily read back and extracted. Um there's information that you can explicitly extract as structured output that we can talk through. There's also information that should be explicitly extracted as tables or charts that should be output in isolated way. So, thinking about how your pipeline actually does this, if you even want to have a you know, simple parsing system that parses all the documents or if you want a structured extraction system that extracts structured JSON is going to be important to evaluate as you look at OCR or VLM or you know, complex agentic models that are used to process your documents. Another thing that I want to highlight is kind of confidence intervals. I'm actually just going to switch to our UI so there's a live demo here. For confidence intervals, what we're looking at is when you extract a piece of a piece of information, how confident are you in the information actually being correct? Because we know for sure there is ambiguous information in a lot of documents. We know for sure there can be conflicting information in documents. Um maybe due to a combination of poor instructions by the initial prompt or due to just straight up you know, misjoined documents that conflict with each other we've seen. Um so, finding confidence intervals is really important on the human in the loop side. Being able to understand where your data is attributed, understanding thresholding so you can tell you know, when you're not super confident or something, how do you go back to the agent and do additional inspection is very important. What I'm showing here is kind of you know, with our structured extraction product and you might be able to build this with your own harness as well, how do you take the output log probs which you know, as your tokens are being generated by your LLMs, there is a probability distribution that exists. How do you take that and use that to drive confidence calculations? Now, you can't always do this especially if your models are thinking models, but that dives a bit too deep into the technical details. But, what I will say is in order to determine confidence intervals and thresholds throughout your structured extraction, you really need to rethink how you are storing the underlying data as well as metadata coming out from your extraction. Just telling someone that I think this OCR output is this piece of text is completely insufficient if you want this level of granularity or detail going through the later parts of your system. And this is important as your agents need to consider you know, how trustworthy certain sources are. >> [snorts] >> You can basically see at 100% everything is uncertain. As you drop it down, certain fields become more or less obvious. And this is also because the LLM is doing some summarization, some conclusions on what the actual document says. You usually don't want verbatim extraction in these cases. Okay, we're getting closer to the actual battleground. So, I'm going to be talking through some of the failure modes that we're going to see today. Um One of the common failure modes that we see is on content density. We've seen this out quite often in production where even for our models, when we get an LLM involved, what happens is that the baseline of one-shotting a document through some kind of screenshotted process or overlaid with the underlying data has gotten much better. But, they do fall short when vision models work based off of information such as page screenshots or the actual PDF layers are missing underlying layered text with embedded metadata. These structures kind of if you knew how these were exactly drawn using for example the underlying lines, you would be able to position much better, right? You're no longer eyeballing what roughly what percentage it looks like. You know the exact positioning at every single X and Y coordinate, um, which gives you additional information. But if you have models that perform super resolution or compression of the underlying images, you will lose fidelity of what's going on in, uh, the actual data. So, uh, to the left I, you know, I have like a revenue and earnings screenshot thing. Um, and to the right I have an output, uh, shoved into Claude and I have an output shoved into Lama Parse. Um, now whether you should penalize one or another is a separate question. Uh, but the failure modes that we do see is that different models will come out with different density conclusions, right? So, Lama Parse tries to go a bit deeper with the revenue in millions and this may be this may very well be incorrect, right? Um, Claude, uh, has rounded it to a much, uh, rounder number, which also may be incorrect. But the conclusion here that people should take away is there is a lot of ambiguity when you try to one-shot complex charts and tables. I'm not saying one is better than the other, but what I'm saying is that there is there's ambiguity that you do do not consider, uh, right out of the box when you have these types of models. Uh, you may be able to fix this by providing examples of what you exactly want ahead of time. Uh, for the purposes of today's demonstration and battlegrounds, I will not be fine-tuning or adjusting any of the prompts to something that we might think we would like. Our goal is to compare just quality as well as kind of what you might or might not want from the models. Um, there is a bit more on recitation and repetition that I'll go through. Um, these are things that you likely will not run into unless you try to build your own LLM harness system, um, but I think these are worthwhile to think about as well. Um, recitation and repetition sound the same. They're very different. Um, recitation and content fil- or content filtering is handled by the different providers differently. Um, I will open a few threads for us to just peruse over as, uh, a way to better illustrate what I mean here. So, uh, um, this is a forum on AI's API focused on why someone's recipe is being filtered out and the API request is failing due to content filtering restrictions. Um, this failure mode is not unique to Open AI. Uh, this failure mode also happens with Gemini. Uh, Gemini has a more formal documented block on this. Um, you can see that Gemini can just stop generating output and say I'm not going to output this to you anymore due to a recitation error. Um, the funny part is Gemini will still charge you for this. Uh, many model makers will still charge you for this even though you don't get your response anymore. Um, so there is a deep question here on how do you handle content filtering, um, and why it even exists. Uh, on the why it exists, I think people understand that, you know, large tech companies are subject to regulation and lawsuits and so it makes sense for companies to not want to uh, process certain types of data. The flip side of it also is some companies will push their own safety initiatives, um, which I think is good and bad in certain areas, but that also means that your content can be flagged and caught, uh, >> [gasps] >> uh, people seem to be, uh, getting caught by these content filters without intention, right? Um, and on overthinking and repetition, I'll just kind of skip through this. The thinking models are especially prone to looping. Uh, your token cost may blow up. All right. Um, we'll go through the live arena now. Um, let's see. So, uh, a few things on, uh, what I want to highlight. Um, we're going to fill highlight a few failure scenarios for common one-shot LLM parsing. Um, we're going to evaluate, uh, qualitatively rather than quantitatively and we're going to be trying to give equal footing to the LLMs without overcomplicating things. Um, in order to do this, uh, in a semi-fair way, um, without doing too much prompt adjustments, etc., um, what we're going to do is we're normalizing the prompts. Uh, we're going to be using no or minimal thinking. There's going to be a focus in our analysis on substantive content over structural issues, um, and we're going to be giving all the models the raw documents as input to preserve fidelity. Um, and I just want to keep keep in mind for people, you know, we're not trying to over-index on single failures. Our goal here is to kind of look over, uh, this LLM as a judge process as well as the output, uh, process to understand what we actually qualitatively care about. Um, I've given a screenshot of the prompts that I'm going to be putting into our system. Um, user prompt, system prompt is right here for people that want to take a look. Um, we are going to be trying to convert markdown, uh, documents into markdown or or, uh, HTML formats where applicable. Uh, and that's just because HTML tables are a bit more representative than markdown tables. They're a bit more expressive. >> [snorts] >> Um, I can dive into the code afterwards, um, but I want to make sure we get to results. Um, you know, as far as a comparison set, we will be comparing LLM output side by side and our goal is to understand the failure modes directly. Um, let's see. Uh, I will kind of load up the Streamlit UI that I've been using to perform comparisons. Um, and this has a set of different, uh, files that have been loaded along with an ability to view the files and directly compare the results using, uh, model A versus model B. Um, and I understand that these models are visually hard to compare. Um, so I've put a LLM as a judge in. Now, be mindful there is some bias if Opus reads Opus's output, there is a bit of a bias in how it understands and parses the documents. Um, let's see. So, I'll get started with a few examples. Um, these are examples meant to highlight differences in model output versus, uh, complex documents. So, I've broken these down roughly into tables, complex charts, as well as, uh, an example of recitation or content blocking. Uh, some of these have been pre-generated. I'm happy to run through, uh, dynamic, uh, comparisons towards the end, but these, uh, generations are specifically interesting because we get to view the failure modes that we see here. Um, okay, great. So, this is an example of, uh, input file that we've used for complex table extraction. Um, and you can say that this table is a bit hard to read and it would not be an understatement. Uh, there are a lot of rows, a lot of columns. There are two tables in fact, um, and we are trying to understand how a model would understand this if you just try to shove it in. Um, you have the output of Lama Parse Agentech. We can also swap this to GPT-4 later. Um, but one thing that I do think is important to note is, uh, on positional encoding and rendering, you can see that, uh, in a harnessed environment, it is a bit more easy to render um, in terms of the actual table itself. I'm going to see if I can run this comparison. Uh, some of these comparisons do time out, so we'll let that run as we do a bit more digging on the underlying documents. Um, I'm going to show this one more time. It's two tables, a lot of rows, a lot of columns. Um, not too easy to visually inspect, so what I'm going to do is I'm going to de-render these and just look at the underlying table output. Um, and one thing that we did notice here ahead of time in review was the number of, uh, table rows, uh, does differ between these models. Uh, you can see in the Lama Parse Agentech model, there are 242 rows. Um, and I will de-render this and then I'll try to look for the Opus model's rows row counts. And the Opus model has only pulled out 152 rows. Um, this is something that we see pretty often with long LLM contexts and dense tables. Um, it's highly likely that certain things will be merged or truncated. Um, now specifically for this document, I don't think we have enough time to dive into the fields one by one to see what was dropped. So, I'm going to drop down to the LLM comparison to see what has happened. Um, as a quick eyeballing of this, it seems like the content completeness is also what it is flagged. Um, it says that model A has more complete data. Uh, model B's output is truncated mid-row. Um, and that's also what we kind of see here. Um, we can take a look at the, uh, detailed results here. Um, let's just pull up metrics and charts. We can see that on a cost per file basis, um, Opus has dived extremely deep into trying to render this document. Um, it's not uncommon for the more capable models to do this. Um, GPT-4 also did this, um, with burning a ton of tokens in a single document. Um, in a harnessed situation, it's more of a fixed cost because you can control the cut-off and terminate early, um, or switch to different model, uh, outputs depending on the complexity of your document. Um, you also see there's like a pretty stark difference in latency. Um, when you have like a structured harness, you can, uh, short circuit a lot of this layout and like understanding what a text is process from an LLM. Directly in a LLM, the latency can be pretty high like uh you have to completely render the input table table as actual input. It needs to think through output tokens and then it needs to transform its thinking process into an actual output. You see this directly here, you know, the latency is quite high across the board. If you were trying to do this in a real world scenario and your your client or customer uploaded like a single document that's just this one table and you took 2 minutes, 3 minutes to process it. The customer is probably already outside of your system or program at that point. Um and there's like a little token usage comparison. I think this is a bit less relevant. What we usually care about is cost per file which can be shown here. Um let's see. So, what I'm going to do here is switch to a few other examples. Um we can go see the output of GPT-4 and now I'm going to be curious on comparing the results of GPT-4. Remind Reminder that last time we had above 200 240-ish rows, I think extracted by Lama Parse. Opus has about 150. Um we're going to take a look to see how many rows are rendered by GPT. Let's take a look. GPT-54 only rendered uh 81 rows apparently. Um so, it may have cut out one of the tables. It may have done something else. Uh we can take a look here. Yeah, it does look like it did cut out one of the tables. So, these are all susceptible, you know, failure modes in different LLMs. This could be an artifact of how we've uploaded the document as a raw PDF. Uh we can also take a look at how this looks like as a screenshot afterwards if we have time. >> [sighs] >> Um let me skip through some of these other scenarios cuz they're honestly very similar. I know we're a bit running long on time. This is a tabular row example. I will do a comparison at the end if we have time. Um I'm going to dive into a security document a security is related document that um we were just looking at. This is kind of like, you know, public reports filed um SEC commission requirements. Um when you do file these, there are disclaimers you have to put into your system, you know, uh determining stock values etc. can be subject to complex calculations. Um and what we found here was uh a few things that I'm going to actually highlight. Um the the input document itself is not that technically complex anymore. So, from a pure textual processing standpoint, a lot of the LLMs are becoming cheaper. It's no longer this kind of outsized spike from Opus or Gemini. Um latency per file uh pretty low. It's not that complex anymore. We're seeing less difference between our Lama Parse harness and the LLMs. Uh you do notice something weird here which is GPT has output $0. Uh we're going to go take a look at the token usage and it also is kind of similarly missing uh for the first document. There's two documents in this data set which is why you see these two charts. Um So, we'll kind of take a look at what happened here with GPT. Um and this is where we're going to highlight kind of the failure modes. You can see uh Let's Let me see if I can move this around. There is a stop reason and the stop reason is incomplete content content filter. And so, immediately you can already see that if you just raw dump your documents into an LLM, uh you do not see great results. Um so, for this one we can do like a comparison with Gemini. Let's see. Uh Gemini Pro preview. Uh these are structurally similar. Um we will see if Opus prefers one or the other. Um and just to remind folks, we're not specifically looking at penalizing single output. Uh we are looking to make sure that overall we understand the landscape a bit better. Um Let's see. All right. So, it is penalizing on things that I don't particularly care about and I don't think we should particularly care about uh which is on page number artifacts. Um this can be easily cleaned up by preprocessing if you were to build this in a real production setting. Um so, we can take a look at page artifact penalization. So, um and the prompt uh given to the judge model is here. Um it's given the source file. It's given the left input and the right input. Um Claude in this run decided that model B included a page artifact of 16, probably not worth our discussion. Uh Lama Parse does filter out footers and headers. Um but, you know, both cop captured the input uh tables and structures correctly. Um let's take a look. There's some complex charts. Let's do complex charts. Um so, there are some complex charts in this data set. Um I'm going to render a few here. Let me see if there's one that's worth digging into. Okay, so here is a complex chart. Um and this is from kind of a mid-year report uh by some investing agency, I presume. Um and you can see that there is this chart that is fairly difficult to parse and understand. If you were to try to put this into your semantic search system, you really want something to figure out what is going on um by decomposing that chart. Um our prompt has instructed our agents to decompose charts where possible. We're going to take a look at the cost per file as well as agency. Um so, in terms of costs, now you're beginning to see that when certain charts are not decomposed correctly or maybe the pages are a bit more straightforward uh straight putting it into an LLM may be sufficient. But, we're going to validate this with the output uh review. Latency is still on par across the board. I don't think anything has stood out as poor or uh too slow. What we're going to do now is we're going to take a look at this results. Everything completed and returned correctly. Uh when we take a look at the mid-year report, uh what are we actually seeing though? Um we see that Lama Parse has decomposed the chart into a table format. So, if you tried to query this with a LLM historically, you could try to get uh some semantic understanding of what you're looking for year by year um value by value. Um to the right here, we have Opus. Um we can do a comparison with Gemini afterwards for folks that want to see Gemini as well. I'll start with Opus for now. Um and you can see that Opus has just completely dropped the ball. Like even though our prompt instructs to convert charts and tables that are numeric in nature into tables, um it has fallen to describing the line chart. And now this can be due to a combination of issues. Um maybe its vision based model isn't able to decompose the table correctly. Maybe it isn't actually using a vision based model but just trying to parse the PDF. All these are possible issues. Um but, you can see immediately, you know, if you just dump it into an LLM and you don't validate the output, you're going to have issues. Um when you come back to search for what did this uh chart represent? We're going to take a look at Gemini now. Gemini does have some really nice vision based capabilities and you do get to see some of it here. Um Gemini, I believe may have additional OCR layer before it actually hits the LLM and so, Gemini often performs better at pure visual tasks. Um you can see that there is differences in how the models treat granularity. Um you can see that Gemini has decided to break it down every 5 years and that may be appropriate depending on what you're looking for. Um the x-axis does represent by 5 years. So, in some sense this is more faithful to uh the axes representations. But, when you try to recompose this and understand the underlying data, I think you'd rather prefer the year by year breakdown because just having the x-axis intersection is insufficient. Um now, to be fair to all the models, um these instructions are not clear. So, judging this as a conclusion that end all is not correct either. Uh but, this is just kind of what you might want to think about as you build in an agentic harness for data loan, you know, data charts and graphic extraction. Uh we'll do a rough comparison here. Um it's possible that Opus concludes that neither are great or that uh Lama Parse has decided to hallucinate data. Um we can also double check if it has hallucinated data just by a visual inspection. Um I'm going to just visually inspect regarding positive negatives. And so, one thing I'm looking for is if there are negatives that are in the data set. Um seems to be fluctuating up and down which is in line with what we're seeing in the data set. Um Gemini also has positives and negatives. So, I don't think there's too much uh to dive into here. Um now, you can see that there is some penalization that Claude thinks is justified for reading order on structure. Um Claude is deciding that the table is fully hallucinated. Sorry if that was a bit of a burst. Um but, you can tell that LLM as a judge comes with its own uh biases. And so, we also have kind of multi LLM as a judge. But, even then, it's possible that something is dropped from all of the LLM judges and they have decided that um a specific OCR based model has hallucinated data when in reality it's more faithful to the underlying representation. Um so, even LLMs in as a judge, even in consensus, can fail fairly often. All right. Um, Murtaza, uh, I'll let you jump in right now if there's any questions or Awesome, yeah. Um, thank you everyone for waiting. I think we'll get to questions. George, there's a bunch of questions in the Q&A section. I'll continue to surface some, um, as they come. But, we'll start with the first that came in and work our way down. Um, give me 1 second here. Here we go. George, I think you can see them on the bottom. Okay, I see. So, uh, we have a question that just came in on how we compute the confidence scores. Um, so, confidence scores are a bit of a tricky thing to compute depending on what you trust and how you've extracted the underlying OCR information. Um, for the purposes of our extraction products, um, which I will jump back to I guess I've closed the extraction tab. I'm going to just go back to extract. Um, our extraction product, there is a confidence score associated with every single, um, I guess extracted artifact. Uh, let's see if this loads. Might have lost my tokens. Um, the artifacts that are extracted have attributions associated with them on kind of where the data or conclusion came from. Um, and in order to compute confidence scores, uh, it is a bit of magic, which is you need to rely on raw logit or, uh, logarithmic probabilities output by the underlying transformer layer. Um, some models do not provide this if they are thinking models. If you are using a non-thinking model that provides its raw output probabilities, you can use this. Um, and this basically determines over probability distribution um, how likely it was going to output this answer or an answer similar to it. Um, now if we find that the answers substantively differ, that is going to have low confidence. Um, if you find that the outputs that we're going to be produced were completely different, that may be a place to flag for a human reviewer. Uh, in terms of confidence scores, there is also kind of combining together multiple sources when you perform, uh, intensive query-based search, which is what structured extraction does over the underlying document. Um, in that case, it adds together the log probabilities and then normalizes it. Um, in the machine learning world, that is similar to just taking together a chain of random events and then figuring out how likely it was to happen. Um, but in this case, the uh, the confidence scores are derived from that calculation. It relies on underlying probability outputs on both the parsing layer as well as the query extraction layer. Um, so, two layers of probabilities that multiply on top of each other, which, um, gives us right now decent accuracy on how likely something is to be hallucinated or incorrect. Um, okay. Uh, next question on how grounding works. Um, in cases where no text is embedded or, uh, text extraction is performed on table output. Um, these can have errors, you are correct. Um, in terms of grounding, we're going to try to look for something that has a bit of grounding in a table. Okay. So, this is an example of extracted data based on grounding in a table. Um, where in terms of grounding, there's positional grounding, which is kind of where it came from, as well as logical grounding, which is why something was concluded. Positional grounding, I think is a bit more of a straightforward question of, you know, traditional OCR bounding box models. This is how we jump back to how we attribute certain pieces of information. Um, so, you know, when I instruct my, uh, uh, let's see. When I instruct my schema to pull out specific information, um, it's going to try to figure out where that information came from from the underlying, uh, textual representation. Now, what we have done in our extraction pipeline is it isn't just a piece of text, it is chunks of metadata associated with their underlying probabilities or logits, as well as their positioning. So, when we perform extraction, we can route things back to exactly where it came from from a uh, positional standpoint, as well as why it came from there from a probability standpoint. Um, and so, there's a bit of a combination here, um, between visual and probable probability-based grounding when you have your conclusions. Uh, do you have examples of complex tables? Is another question that's come in, uh, three-dimensional or transposed tables. Um, yes, we do have examples. I can run through some right now. Um, we also have benchmarks that run through this. Um, our sales team also loves demoing this cuz it is honestly not a very straightforward task for people to even do by hand. Um, I'll see if I can pull one up here from our examples. If we don't have one, we can follow up separately for this question. Um, let's see if we have one here. Um, oh, this is somewhat of a tabular table. Um, I'm going to see if there's a bit more of a complex, uh, table that does decompose, uh, uh, fairly well. Okay. Uh, this is going to just fail across the board, so I'm not going to necessarily, uh, demo this. We can talk about this separately. This is an example of, uh, what a bad table would be that your customer could give you. Um, if you dump this into a vision-based model, it would absolutely compress this into being useless. Um, but we can see that with, uh, visual grounding, there's better outcomes. Um, what might be more interesting is a comparison of this document, which is, uh, tabular and has empty spaces, uh, has joined joined headers, etc. in nature. Um, we'll just go through this real quick as far as a comparison. Um, I will Well, we already have Haiku pulled up. We can take a look at, uh, LLM-based comparison. Uh, and sometime sometimes these simpler models do perform a bit better. Um, sometimes the complex models do overthink. Um, what we will do a quick comparison here is kind of how do we merge these, uh, header components? You can see that there are there's a complex legend here combined with, uh, multi-row joined headers. So, this is not super straightforward to represent in markdown. Um, we can take a look at how Haiku has represented this as well as Agentic Plus. Um, you can see the legend is pulled out. There is some confusion, uh, in how the legend should be encoded into the table itself. You can see that, uh, the hospital, uh, ambulatorio, well, this is also multilingual at this point, so, uh, translation may not be great. Um, is pulled out on top. You can see that it's pulled out on top. There is an isolated section for how this is joined into the table. Um, I'm not going to dive into what we think here. Um, I'm going to let the LLM as a judge work, uh, work on its own here. Um, So, uh, this is Opus judging the output of Haiku and then parse, uh, uh, parses Agentic Plus mode. Um, you can see that there is, uh, issue with hospital groupings not being preserved correctly. Uh, reading order has gotten joined. So, what this probably means is if we look at the source document, um, there is Well, this is actually is not great cuz I do not understand Spanish. Um, but there is some aspect of hospital grouping that has gotten garbled, I assume, uh, from either one of these two rows. Okay. Um, let's see. There are three-dimensional tables and transposed tables that I do not have pulled up right here, so, um, we'll get back to you on that one offline, uh, for the for that specific question. Um, we can jump through other questions, Murtaza. Okay, for the light parse release, is it possible to structure the output in JSON or only markdown? That's a good question. Um, I believe right now what I've seen has only been markdown. We do have JSON capabilities. Um, I will need to double-check with the open source repo if that has been released into the open source site. Um, outputs in JSON on the visual grounding side, as well as the bounding box, uh, coordinate side are very useful. Um, the there are certain things that you cannot get out of the structured output in JSON with the open source model, uh, such as certain aspects of probability, um, since there is no, uh, language-based model doing transformations afterwards. Um, but I believe if it's not there right now, structured JSON should be there at some point. Um, probably is there right now. Let's see. Okay. Um, we have another question, which is with chart understanding. Um, there is a problem on how we can actually trust charts that are output by LLM's. Um, it does not give straightforward bounding boxes. Um, we have seen similar requests to having uh, at Lama Index, we've seen similar requests to, uh, character-based, uh, bounding boxes rather than, uh, string, row, or, you know, word-based bounding boxes. Um, so, bounding boxes are a good question on like how granular you want specific bounding boxes. Um, as far as chart understanding, there is a way to extract confidence intervals based on LLM output. When we do combine specific charts and tables with LLM understanding or LLM parsing, we're able to get fairly accurate or along a distribution fairly accurate confidence intervals. The LLM does take into consideration stuff such as ambiguity or kind of poor scanning, poor encoding. So, if we just give it a specific cutout of a table or chart to manage, it usually does a pretty good job with identifying confidence. And we can't say that we're 100% confident. Usually, we have users kind of flag a threshold that they're comfortable with. And they will perform additional human in the loop review if they want. But we get the same issue also with non-transformer based models. So, even in the old school IDP OCR world, there's a level of trust that you need to have with your capabilities. The nice part here is in the new world, we actually get confidence intervals. If the models So, there's another comment which is if the models could judge this without ground truth manually provided to it separately for testing purposes, we wouldn't need a harness. Let's see. That is right. That is a good That is a correct conclusion. If the models could judge, it could also generate. Although, judging has always been an easier task than generating. What's the main difference between parse mode and extract mode? That's a good question. We get this all the time, which is like why is there parse, why is there extract? The exact reason for why parse and extract exists is specifically so that you can have structured data. And when you do have structured data, you can you can be not faithful to the original document in the sense that you can summarize documents, you can synthesize more complex information. If if you're trying to figure out the country that something happened in and your original document says that it happened in New York, you can conclude that it happened in the US, right? And if you have a straightforward parsing implementation, you would not be able to directly parse that out of the document because the underlying document says New York. If you're trying to be faithful to the document, what parsing does is literal transcription as faithful to the document document as we can be. Now, you can provide a system prompt to adjust the output. But usually, what we want is we want the original document as faithful as we can have it. That's parsing. Extraction is when you want to begin uh in a semi-one-shot manner getting this complex data out and having some grounding associated with the data itself. So, by combining the two, you know, the parsing step first and then doing probability calculation and synthesization or synthesizing it, you get extraction. Yeah, hopefully that answers your question, Dustin. All right. Um is There is another question which is is there a way to give some confidence to LLM response? Not necessarily this the stability like uh the recommendation is Let's see. >> [sighs] >> Sorry, let me read this a bit more. Is there a way to give some confidence to LLM response like the How do you quantify critical? >> [sighs and gasps] >> Okay. I So, I think what this question is asking is how do we give uh more grounded confidence to the actual LLM response? And how do we actually quantify what is useful insight from the LLM or not? Um I think there is There are two things here. One is what type of insight you're looking for. If you're looking for the LLM to synthesize and summarize, that's one thing. If you have structured output, that's another. If you have structured output and confidence like you do with extract, I think we've already explained how the probability is calculated by synthesizing from the underlying LLM LLM's probability distribution as well as concluding where the data was pulled from, multiplying together that probability. Um In terms of giving more confidence to the LLM response, that is one of the major differences between our Agentic mode and Agentic Plus mode and is also likely a major difference between the LLM providers' more complex modes and simpler modes, which is how deeply they think and self self-validate. The LLMs have much less of a harness to perform self-validation. So, if you let a LLM model think for too long, it can wander off in the wrong direction. And so, if you let a model think a bit more, confidence doesn't necessarily improve, accuracy doesn't necessarily improve. If you do give a LLM a better harness to kind of germinate and think a bit more on a document that is provided to it with the right tools or the right restrictions, it usually can self-improve. That is one of the things that Agentic Plus tries to do over Agentic, which is that self-reasoning and improvement process. All right. Next question. Does it capture So, this question is on if the parsing process captures relationships or answers cross questions from one PDF to another? So, the the native parsing and extraction product that is offered open source through Lama Parse, etc., does not handle Q&A across different documents. There is an index feature, which does allow users to have, you know, sets of large files inserted into the system for Q&A. But out of the box, there is no cross-document capabilities or kind of you might think of as long context search used in our parsing and extraction process because things are usually isolated. But if you build a indexing system or you have a longer system for rag you can handle much larger document data sets. For our purposes today, we're just talking about parse and extract, though. All right. Next question. Oh. There's one more that just just popped in before I was about to end. Okay. One question is what were were are the hardest challenges for our team while building Lama Parse? Good question. There's a lot between just going zero to one and going from one to 100. Or in our case, I guess we've processed like a billion pages now. So, So, whatever that scale is. In terms of going from zero to one and just having something functional, a lot of the problems that we had with structuring Lama Parse was breaking apart this traditional OCR thinking process into something that could uh compose, I guess, output that would be useful for an LLM as well as creating this harness for an LLM to be able to iteratively think without going off the deep end. So, we have like very very much, you know, deeper evaluation and benchmarking capabilities than just doing a visual comparison here. Everything that we do to make sure that the harness makes sense and the harness actually improves the Agentic process is difficult and I think is new engineering that has not occurred in industry until the last 5 years. And then there is the more traditional work of scaling it up where, you know, you're dealing with more traditional infrastructure and systems processes. We do have a blog post coming out And some of those issues I've discussed today, too, which is with white space repetition as well as with recitation errors. When a LLM provider has decided to refuse your query, what do you do? And if an LLM provider has decided to go crazy with returning you infinite white space strings, which can still happen in the more advanced models today, especially if you put raw binary into those models how do you terminate that? How do you handle the cost of burning max context tokens on a advanced model when it begins outputting garbage? So, understanding how to handle LLMs has been another another learning lesson for the team. Okay. We have one more question, I guess. And this question is on the Q&A process for uh any documents that are processed and then answered by an LLM. So, if you have a chatbot or something an LLM can respond with a recommendation. But how do you quantify and then validate recommendations are correct? You know, when a LLM just says there is insight, how do you actually give it confidence that there is insight? In a traditional rag or rag or grounding system, if you do not store the probabilities of how confident the system is that something is true, I think all you can do is perform model validation and deeper thinking on the concluded documents that are cited. So, we do have systems in place that will do stuff like perform an initial rag search, look for documents that are similar to a specific topic, and then you can go into the deeper kind of agent logical thinking loop of let's get that document and let's dissect it or let's get the text of that document and dissect it and then producer and conclusions based on the groundings of a document. This goes a bit deeper into kind of the deep research products that were launched probably a year or two ago by the different model providers. For the purposes of our discussion today, I'm not going to dive too deep into how you would fix fix confidence or determine confidence there though. Okay, perfect. For the sake of time, I'm going to call it George if that's okay with you. That's fine. Before we log off George, if you go to the next slide, for the folks for the 80 90 folks who have still hung on, feel free to try LlamaIndex for free. It's free to use. Go to LlamaIndex.ai or go to LlamaIndex.ai and then find the button to sign up there. You get 10,000 free credits. We'd love to get your feedback. We have a Discord. All that type of great stuff. Thank you for staying till the end. This recording will be sent. George, thank you a bunch. Hopefully this was informative for everyone and we hope to see you at the next one. Thank you again. If there are any more questions, you can always George our join our Discord and ask there. Our team is very responsive and very very active. Almost too active sometimes. So, join LlamaIndex community, keep the conversation going and we'll see you at the next one. Thank you all again.