Inside ParseBench How to Evaluate Document Parsing for AI Agents

Key Takeaways

Welcome everyone. Good morning. And thank you for joining us today. My name is Gautami. I lead product marketing here at LamaIndex. And today we're going to dive into ParsBench, our open benchmark for evaluating document parsing for AI agents. Some quick housekeeping notes before we get started. The session is being recorded and we'll share the link afterwards. If you have questions, please drop them drop them in the Q&A box and we'll get to them at the end. So without my further ado, today we are joined by Preston Carlson. He is a member of technical staff here at LamaIndex and was one of the core researchers behind ParsBench. He was deeply involved in the methodology, the metrics design, and running the benchmark across multiple parsers. So Preston, the stage is yours. >> All right, thanks everyone. Here we go. Thanks Gautami. Yeah. Um Yeah, so I'll be manning the station here for next half hour or so. Um Yeah, so the agenda for today, we're going to discuss like at a high level why document parsing accuracy is hard to measure and how ParsBench was designed. The bulk of what we'll discuss today will be the five dimensions that we're using to measure parsing accuracy and diving into those, and we'll give a note at the end on how to adapt ParsBench to your use case. Um we'll have time at the end for some Q&A. Um Gautami will be like aggregating your questions along the way, so please do send them as they as they come up. All right, thanks. So yeah, kind of the the state of the art before ParsBench, um there have been Oh yeah, we'll let me share a video link at the end. Um, things like that. Um, yeah. But, yeah. Uh, state of the art for for PartsBench, before PartsBench, um, there have been quite a few document parsing benchmarks, um, but most of them like predate the agent workflow era. So, like they're good enough for human in the work human in the loop workflows, but not really reliable enough for an agents to act on that doesn't like have, you know, human intuitions to go and double-check a lot of things and it's like often gets things in context once unless it has time to retrieve them or um, is a lot more like limited in its judgment. Um, and a lot of pre-existing benchmarks also like just weren't weren't as thorough as they could be. Um, like getting a huge corpus of documents and having and test to run against them is expensive and it's it's hard to get ground truth. Sometimes they there's ambiguity and sometimes it's just too much to do exhaustively. Um, we'll see this with some of these dimensions. Um, but that meant that a lot of the like pretty much all of the existing benchmarks didn't really meet our needs in trying to push the envelope of document parsing accuracy. Um, core issues here are like the documents themselves are just underrepresented of like modern workflows, especially in enterprise. Like, we're not seeing enough financial documents, just weighted a little bit toward like academic articles, that kind of thing. Definitely useful, important, but like a very small in the space of all possible PDFs. Um, and then the metrics aren't haven't been that well calibrated. Like, pre-existing, um, metrics for like chart [snorts] parsing and certainly table parsing, like the state of the art is something called TEDS. Um it treats a table as a tree. Like it's easy to calculate, but tables aren't trees. It kind of doesn't really match your intuition, and it's it's hard to use to really improve your your parsing performance. And often you just have these like pre-cropped inputs that are totally divorced from the full page context, which is what you actually see. Like pre-cropped inputs are kind of assuming away a large portion of the work, which is you know what is on the page and where it is. Um Yeah, and so in developing ParsBench, we'll go back to that in a sec. We gave all of our documents like full page context. So not like pre-cropped tables on a page, like the full page itself, so you have to identify like how the table is present, where it is, and what the actual content is. Um And we designed some of our own metrics and used some pre-existing ones, so that they're tuned for agentic use cases. They have that reliability we need for workflows that are not human in the loop. And it's built on enterprise docs. So we like have some academic docs in here, but also like financial and insurance documents. Like all all over the place. And you you see a lot more diversity there because there's not the like a filtering function of peer review that makes things a little bit more legible and and easy to understand and well formatted. Um importantly, these are also like actual PDF documents. It's not just a visual benchmark. A lot of pre-existing benchmarks are also just just images. Um and you lose a lot of useful context when you divorce it from being a like native PDF. And so ParsBench is just over 2,000 pages. A lot of a lot of rules here so that it's deterministic to compute whether a um whether a result is actually meeting the the requirements. We evaluated quite a few methods and we keep evaluating more, especially with community support. Um and like all of this is like available and reproducible. We have GitHub GitHub links at the end. Um we have a a website for this if if you haven't seen it already. Yeah. Yeah. On to the on to the bulk here, the five dimensions. So, the five things we really measure parsing performance against are charts, semantic formatting, visual grounding, tables, and content faithfulness. So, going into these one by one. Go charts. The target for chart parsing is structured data, not description. So, right here we have a like a chart on the top and an example output. We This is like incorrect output on the bottom. See if we can I don't think we can zoom in here. Um But really what the the charts dimension tests is like whether the chart data comes out as a usable table. Um and like what this matters is that an agent can't compute over a descriptive data. And a lot of the descriptions you're going to see are pretty ambiguous. Like, how would you describe this accurately? Like, it's really complicated. You don't want to Oh, yeah. Yeah. You don't want to like assume away a lot of the a lot of the relevant detail, but you also don't want to assume more knowledge than you already have. So, like for this chart here, like what we consider a failure here is like kind of a list of key values here. It's divorcing a lot of the data points from their actual values. Like, it's just essentially listing all of the like pieces of text that are in the chart. Um but what we What is a lot more useful is having a like structured data representation. In this case, it's a table. Um And you'll note here like the headers don't perfectly match the text that is listed on in the legend here. That's totally fine. Like we we allow some flexibility there as long as it's like semantically very close. Um the important thing is that you're able to like reproduce the data that is definitely in the chart here. Um We measure chart accuracy using a like a custom metric here we call chart data point match. We do we define this pretty thoroughly in the code in GitHub and in the parchment paper. Um But at the core there at the core here is like we annotate up to 10 data points per chart and we make sure that they are like unambiguous data. So, if we see like a really continuous line chart but there are like some labeled points, those are the things that you should be anchoring on here. Um this isn't just because it's like harder to match the continuous line. A lot of it is we don't have access to the underlying data so we don't actually know that the true granularity. That's a pretty subjective what it could be. Like it's not always listed whether it's per day data or per hour if it's like market market volatility data. Um So, making sure that we're getting the data that's definitely in there and it like referencing back to the like source document which we'll see in other dimensions. Um So, that if we need greater granularity than these like 10 data points can provide, then a user like either a a human going through the agent's results or the agent's looking through the parsed results can go and cross-reference and dig deeper if they need to. Yeah. But we we make sure that we capture like the series name, the access category, the chart title, and that a data point passes only if the value is correct and all the labels are present. So, we can't be missing like a whole axis or something like that. We have now semantic formatting. This is another dimension here. Um is where formatting is like relevant to the understanding of the document. Um this says whether the parser correctly preserves formatting that carries meaning. Things like strikethrough, um bold often, not always, superscript, subscript, code blocks, and LaTeX. Um And this this matters because when formatting is stripped from the output, agents lose signals that indicate what content means, not just what it says. So, here on the right, like a lot of the formatting isn't especially relevant. Like the italics here in the sidebar don't matter all that much. But this like header hierarchy does matter. This thing like referral, this is like some level of header over this list. Likewise here for initial contacts and so on. And so an example output like this, we see that yeah, the headers are correctly marked. We've got like this top-level header. Um Actually, we have we have a good question here that I want to answer. So, somebody asked is is ParseBench evaluating different parsers using the framework or here itself is the parser? ParseBench is a framework for evaluating parsers. Yeah. Like Llama Parse is our parser that we have evaluated and and we have many competitors and open-source um and parsers that we that we've evaluated as well. But yeah, sorry, back to the semantic formatting here. So, yeah, header level one, we have like infant toddler program, and then we have successive headers. We've got like the first steps is a header level two, and below that is referral. This is important for us to capture. But, then there are other aspects of semantic formatting. Like, we It's important to hear that Like, in this end, we'll see a related dimension content faithfulness that like reading order is preserved. And like whether this is bold here on the right like like could or could not matter. Yeah. Um but the the things that are that the semantic formatting is like really targeting are things like strike through here where like price revisions if you strike through this and you in the original document, but in the end result, you remove that, it is completely ambiguous what the intended interpretation of this should be. So, it's really important you capture this for agentic workflows that cannot go back and view the source document immediately or where like where it's expensive to do so. And for like sub and superscript, here like if we have a if you have chemical formulas or equations or footnotes, it's important that we are able to like preserve that formatting so that we know like here it's maybe inferable from context, but it can become very ambiguous with equations very quickly and with footnotes, you can just lose all all context if it's unclear where this one is being referenced. Okay. And then with things like rendered math, this can this can become a mess real quick because there are a lot of different like source equations that can be incorrectly mapped onto the same completely flattened text. Um and code blocks like for readability, you definitely want this and it in like this like Python, if you have a code block and you lose all white space, then you also lose the semantics. Yeah. Yeah, and so we evaluate four properties here. It's like text styling, title accuracy. Um this is like the different title levels, the headers. LaTeX rendering and code block preservation. And then we we wait this Um we we go over the exact waiting in the paper. Um the important thing here is that like text tiling and title accuracy are like extremely important. Um LaTeX and code blocks are weighted lower um in large part because they are don't appear as much in documents. [clears throat] And when they do, they they matter, but not as much as like correctly capturing the hierarchy of information in the document, especially when feeding it to agents. So, visual grounding here, this is our third dimension. This is for like every parsed element, we want to find like what it is and where it is in the page. This is like really important not just as its own benchmark, as we often see, um but as part of a full document parsing benchmark. In prior benchmarks that focus on like just crops of a page, this isn't really relevant. It's that work is kind of done for you. Um like whether you are using this like visual grounding before you go and parse or after to attribute your results, like this is important. Um what this test is like not just like that you can attribute it accurately, but also precisely. So, in this example here on the right, you can see like both of these are like kind of defensible. They they seem pretty close. So, like here you can see like more granular bounding boxes over elements. You can see these are images, this is text. Like this big gray block here like also covers these things, but if you are looking for a specific text element that's attributed to this, like this is more precise and easier to to look at the exact text. Yeah. Um this this is really important like for auditability. That's like the the number one Yeah, the number one use case we see this is relevant for, but it's also helpful like just moment-to-moment when you have an agentic workflow being able to go have the agent site information or like find other relevant information near what they were what it was looking at. Yeah. Like here's an example of like what we probably want on the left, like this more granular attribution, and what we wouldn't quite want on the right. The thing to focus on here is not just the like tight bounding boxes on the text on the top, but this this big block of text on the left. Like if we are looking at a specific sentence in the output and we want to and we want to find like where in the source document it's from. Like it's it's not wrong and it's not completely unhelpful to see that it's in this huge block, but there's still like a ton of work left for the human being if in locating exactly where in the page this is from and and verifying that the quote is from here. If we can say, "Oh yeah, it's definitely here." We can very quickly tell whether something is being hallucinated or if just the the evidence is backing up the the claim here. And so we measure visual grounding at a at a high level. It's pretty simple. The the full math is in the paper. But like we need to locate it correctly on the page. We need to classify it correctly. So not just you know, that it's here, but that this is text and we're not somehow thinking that this is like a table, that like this is an image, this is a chart and not like a signature. Signatures get misclassified as line the line charts all the time. Um and we need to attribute the right content to it. So it's great to find all these bounding boxes and decide that yes, this is text, but if you have like two similar paragraphs and you attribute the output of one paragraph to and you usually like swap the attribution of the resulting output, then it's still not all that helpful. Like you need to make sure that you're accurately mapping the two onto each other. And then the score as as it is for the other things is a proportion of elements that pass all three checks. Yeah, um we have a question here about using parts bench when your domain is narrower than general case. We actually have a bit on that at the end uh in adapting parts bench. So I promise we'll get there. Yeah. Yeah, so tables. This tables dimension is a bit near and dear to my heart. This is the one that I worked on the most um in the in the parts bench paper. So a lot of Yeah, the the core thing that this test is whether values are attributed to correct headers and not just whether the table like looks structurally correct. And the reason this matters is that like structural errors aren't just like a swapping of two cells. If it's in the header for instance, then it completely messes with the semantics of the rest of the table. Um like the little little small here, but we can see that like if we have a financial table and we swap two cells, essentially just two characters where like 2003's data becomes 2004's and vice versa, then we are just completely wrong in the output. Like structurally, this is a pretty good parse and like you need to like give it some credit for that, but like if the semantics are completely messed up, we need to to penalize this pretty heavily. And like pre-existing metrics do not do this. Pre-existing metrics mostly treat headers and data cells as like the same kind of thing. They don't think about the overall semantics of the table. Um we talked about TEDS a bit. There's another um like free structurally oriented metric that we is very good for structure called GRITs that we fold into our metric. Um, but we um, like equally weight grits, just essentially how close is the um, parsed grid to the what we consider to be the ground truth grid. And table record match, where we consider the semantics of the table. We'll go into that a little bit more in a sec. Um, and see how how close these map. And we have a pretty strict requirement for that. Um, yeah, actually we'll just dive into this here. So, for table record match, like consider this table in the upper left corner. Look at this as our ground truth. We've got a bunch of companies, we've got their revenue and their year-over-year growth. And given these headers, like this is essentially a bunch of records. We've got like collections of values. We have like the company Acme has revenue 4.2 billion and year-over-year percent growth of 12%. And we want our prediction to match that. It is in the like semantic semantic world here, it is okay to reorder columns and rows. Um, this is not true for every table. This is true for a lot of tables though. Um, and so table record match we use for like a large bulk of the the tables like that in in conjunction with grits. We use these for the bulk of the tables in ParsBench. Um, yeah, and so what we do is we take the prediction, we kind of convert that into its own set of records, like with companies, revenues, and year-over-year. Um, and if they have extra columns, it's penalized for having this like irrelevant information, but it's still but it still gives it like some credit for getting most of this information correct. Um, and we have a pretty strict requirement on whether the data matches. We do a bit of normalization, so you know, if you have a space between the dollar sign and the and the number, then you're not penalized. But, if you get like a single digit wrong, then you're still going to get like in Yeah, you're you're you're still going to be penalized. So, like the year-over-year percentage for Beta Inc. should be 8%, but it's 6%. It gets like zero zero points for that. Like it needs to be exactly correct um for table outputs, a lot of I'd say the majority of workflows that expects tables expect that information to be essentially perfect. Um unlike things like maybe text output where like small deviations you can mostly make up for it because you're not depending on like an exact value in an exact cell being exactly correct. Um here like you will just make incorrect um financial analyses if you have the incorrect value here. So, we have really strict requirements Yeah. Um And here like for table record match, this result gets like about like a 50% score because it like misses this value, but it also has a bunch of like extra information that's just totally wrong. It's got like an extra um record here that it shouldn't have, and it's got like a a region column that doesn't exist. Yeah. Yeah. Um final final dimension here is content faithfulness. This is related to the like earlier semantic formatting. It's like an overall property of the document. It's not for like just specific elements. And this tests whether extracted content is a faithful representation of the source. So, like you're not dropping sentences, you're not hallucinating content, or messing up the reading order, and you're not like interpreting the source text in some way. Like, if you for a given image, like you don't want to just be providing like a bunch of extra descriptions um for these here. You don't want somebody to be like describing or you don't want the output to be a description of the page. You want it to be a transcription of the page. So, it's like essentially, if you looked at the source document and you looked at the output, does it match what you would expect if you were to like turn it into mostly like unformatted text with a little bit of formatting from Markdown. Yeah. And like this this matters because a parser can look really really accurate, especially like to the naked eye even when it's at 95%. It's still be like dropping clauses here and there. Um and if modern LLMs are the in the parser at all, there's always the possibility of like completely hallucinating some content. We want to be catching that. Um text similarity doesn't quite doesn't quite catch this, um and isn't as as strict as we try to be. Yeah. Um so, this example here, like you can see we have like the content for like the three chapter header. It starts, and then it starts interleaving with four, and then it starts with number five, and then it like duplicates a bunch of the content from three as a table. Like, this this is not a faithful representation of the source document. Like, we we have some additional examples here of like this is a like a little dense, so it's hard to see fully, but you can see I think the the reading order here is messed up. Yeah, Department of Energy, Federal Energy Regulatory Commission. It's Yeah, just kind of all over the place. Um this here Oh, man. Yeah, I think this one is like in large part fully like hallucinating outputs. We've got like the reading order's messed up. Like inputs should be then listing all this information materials and it's Yeah. Like the these are pretty egregious examples. Yeah. The way we measure this specifically is whether the text is correct, whether it appears in the right reading order. And we weigh text correctness about twice as heavily as reading order. Um a lot of the reason for this is in like multi-column layouts that we saw here, um it is like it is like mhm it happens a lot more often than one might expect that the reading order is messed up. Like a given PDF can just have a really exotic format and the the text can end up in the wrong order. Um but downstream workflows, especially agentic workflows, um are actually pretty robust to reading order mix-ups when you're like feeding an entire page's content um into into an LLM. Um it's it's I would say it's actually more robust than a lot of like human in the loop workflows would have been. Um it's important that all the content is there and the reading order still does matter quite a bit. Um but if it's like huge blocks of text and you just like mismatch those a little bit, it's like somebody easy to infer from context what the what the original meaning was. Yeah. Yeah. Um You can look look questions. Yeah, we can get to those in a minute. Yeah, overall what the results from the from our tests and like the community tests have revealed is that like chart is the most polarizing dimension. Um they are like images that you that have to be interpreted into data and you kind of need specialized machinery for this. Like only five of the 14 providers we benchmarked like exceeded 15% on this and most specialized parsers scored below 6%. Um just like essentially just doing OCR and no mapping into some structured format. Um semantic formatting is something that a lot of like almost all parsers do pretty poorly on. Um there's like just a huge huge range. Um sometimes it's just simply not supported. Um and sometimes it's really requires this full page context and when you have been testing on like these pre-cropped inputs, you're not going to have that full page context as much. And often you are you benefit from having access to the underlying PDF information. Yeah, um visual grounding um exposes single pass VLM limits. So two of the major VLM providers score below 10% and I'll I'll note here like different LLMs have different like screen spaces, they have different um coordinate systems they use for making bounding boxes. We did take this into account. So we like for each of the different providers, we made sure that we were actually matching what they natively output. Um we didn't just treat them all the same and we we still saw like a pretty big gap here. This is something that you're not really able to one-shot without a lot of additional work and certainly not in like the more exotic layouts and like really really dense layouts. And lastly here, content faithfulness is table stakes. Like it it's great if you can, you know, parse a chart correctly, but if all the rest of the text in your document is bad, then it doesn't matter. It's not really useful. Um Like a most parsers do pretty well on this. This is what the like state of the art for many years has like on and focused on even before LLMs. And so, the baseline here is pretty good. Um but it's still still not perfect. Yeah, and last thing here before we get to questions, um I want to talk a little bit about adapting ParsBench to your use case. Um I was also going to actually ask this. Um Uh you use this one one uh person's question here. Um I think Steve asks, "How do we best use ParsBench when our domain is much narrower than than the general case?" "For developing solutions for the insurance industry, is ParsBench the right wrong or right tool?" Yeah, this is a great question. Yeah, so we deal with a lot of insurance documents. And in that case, like you're right, you don't need to care about charts. Tables matter a lot. There's a lot of diversity in tables there. Um like maybe visual grounding matters less. Um the documents themselves are better like more consistently formatted, often like native you know, PDFs not not scanned as much. Um so, this is like a maybe content faithfulness, semantic formatting we run into, and and tables definitely. Like And so, based on your domain, you can like choose to weight these different dimensions. Um Yeah, just weight them differently. Like in financial documents, like charts matter a lot, so do tables. For citation and verification workflows, like legal visual grounding can matter a lot. Um And for like general document processing, if you don't really know, like we we tried to give this a a pretty a pretty good distribution of documents and dimensions here. Like we we want our we wanted our base score that we provide to be useful for you here. So, like that that's a pretty good place to start. Yeah. Yeah, finally, before we get to questions, like this is open source. We made it reproducible, and we want you to contribute. So, please do like visit these sites here. Um, are we are we sharing the the deck with them? Go to me? Yeah, okay. We'll be sharing the the deck as well, so you can like click on these. But, also if you Google Parse Benchmark uh very very quickly you like have links to most of this stuff. Yeah. >> So, uh so we have questions. I'll start with the first one. So, this Are you using Yeah, so you use Parse Benchmark after using Llama Parse for each document to find the accuracy. That's one of the earlier questions that I think you missed. >> Oh, yeah, great. Can you mute yourself please? Okay, great. Um, yes, use Parse Benchmark after using Llama Parse. Um, yeah, I would say the the general Mm, okay, yeah. Um, this is not intended so much to be used like I guess like on right [clears throat] after you uh parse a document using our product. This is like if you have a like if you have a use case for Llama Parse, like I would say go look at at Parse Benchmark. We have like kind of this suite of documents we've looked at before. There's like 2,000 pages. Um, you want to look at the scores on those. You can definitely like make your own ground truth, make your scores for each of these dimensions on your documents, but from experience it can be quite a lot of work. We wouldn't expect you to do that. Um, this would be like if you're evaluating us and other providers for parse use cases or like open source models to go go look at that. Hope that answers the question there. Yeah, like like in essentially in order to To the accuracy on your own documents, you will need to like come up with what the actual end result should have been on your end and then and then you can evaluate it. Um but we the the point of these benchmarks is to give you a sense across like documents in general. Um and then you can weight it to mhm suit the documents that you're going to see. Oh yeah, and then we got uh question about like images and tables. Um we don't cover that very much in Parsel Match. Um we cover it a little bit. I believe we like check whether the images are like in the correct cells. Um there's like a lot of there are a lot of possible like valid acceptable um interpretations of an image um especially in in a table and a lot of the like enterprise workflows don't have these images in tables. So we did not not focus on that for this for this benchmark. Um yeah and we we felt like we especially with like the the documents we've seen um and the variety of different ways that parsers decided to output images in tables specifically, we couldn't like fairly evaluate everybody against each other without like waiting it toward toward us or one of the major VLMs. Yeah. Yeah and then like whether Parsel Match can be installed and run. Yeah, like I recommend going to the to the GitHub. We have instructions on that there. Um Yeah. And oh yeah, great question here about uh content faithfulness. So did you notice if closed source or API-based models fail mainly when text is small or dense due to a resolution bottleneck? Is this an advantage for open source parsers in transcription? Um yeah so content faithfulness yeah the the two major reasons that things failed, one definitely just density. Um sometimes it's resolution. Um but even if like the resolution is supported, sometimes it's just not what they were trained on and it's like really hard to generate just a sheer amount of output correctly for these smaller models. Um the other reason actually is reading order. Like there are some documents that just have really challenging reading order. If you look at the source documents that we have, um they're in the the GitHub, Hugging Face, or um the the ParsBench website, you'll you'll see that there are just some really challenging ones. Um even like old newspapers, which like humans are pretty used to reading newspapers at this point, um even though most humans aren't reading newspapers proper day-to-day, like it's easy to for us to kind of figure out the reading order. But when it's really dense, um and there's a ton of columns and different articles that are related in interesting ways, or a lot of in like internal financial documents, people get creative and it's it's hard for models to get the the order right. Yeah. Um Does Does ParsBench include tests of multi-page tables? Um often working with Excel files exported PDF the table is split. Um ParsBench right now does not. This is definitely something we're doing in an extension here. Um we we definitely see multi-page tables and it's something that we support as this merging across pages. Um Yeah, like there is a like a large variety of cross-page tables and for ParsBench like in general we focused on like single-page content. Um there's like expansion from the pre-crops to single-page and we like are definitely moving toward like full doc and multi like cross-page content. I'm getting a lot more questions here. Great. Let me see here. Um I'm just going to go through these in order. Does that make Look good on Doc Jim? Okay. Um, for every of the five axes, should they wait equally important when downstream utility is concerned? Like knowing if something is cursive might matter less than just getting a table partially right. Um, yeah, I I think we addressed this earlier. The big thing is like you should adapt it based on like what you the documents you're actually seeing and you're right, like based on the actual downstream task. Like there are some cases where like detecting whether a signature is on the page and whether the signature is like actually written or if there's just like an empty signature block is like paramount and there's some cases where you just happen to have a signature in the documents that you're scanning it doesn't matter. Um, yeah, like it you should definitely if you have a specific use case in mind, you'll you'll want to wait these differently. Okay, yeah, somebody's asking Um, we are a large New York City construction company. Um, worked on creating custom document ingestion pipeline with chunking vector store and graphing in combination. specifications drawings. Um, so yeah, I the the question here is like whether Lama Parse is enough for that with the they should be using ParseBench as well. Like I would say here like they're they're just kind of serving different purposes. Like a Lama Parse is the like the product the parser that would do this injection for you. Um, ParseBench is like what you can use to see what like which providers meet your case. Yeah. Like we we give you like some sense of accuracy. Um, but it's not like a it's not a production document ingestion pipeline. That's what that's what Lama Parse is. Um, I think we answered this table embedded with images question. Um, yeah, we've got we've got some like Lama Parse specific questions I might defer for for later. These are good questions. Um, but I want to make sure we're getting to to the ParsBench questions. Um, like but definitely like you can store parse results and query them however however you like. You just need to make sure you download them. Mhm. So, in comparing well-known parser options to build an ETL workflow for financial documents, how much should I weight my choices based on published ParsBenchmarks versus my small representative data set data set? Well, like depends on how small your your data set is. Um, like if I I would say when evaluating against your own use cases, if you have like really a really homogeneous, really similar set of documents that you're parsing, it's okay to have a small representative set and then parse against those and see if it meets your needs. Um, like cuz it in making a general document benchmark, it's simply the case that like it's not going to exactly match everybody's distribution and like making sure that it works for your needs is pretty important. Um, I'd say like the general results will give you a good idea of like how we perform across documents in general, but if you have like really specific needs, like I would say even add your own tests on top of these. Yeah. And we have very complex product data sheets. Is this also possible as a benchmark? Yeah, I know. We we do have data sheets in this benchmark, um, especially for for the sake of like tables. Um, don't know what we do for charts here. Um, like that is definitely something to evaluate us on here. Um, I think you can >> [clears throat] >> also go through and see in like the ParsBench I don't know if I see it in like the the ParsBench website, look at the pages we have, see if these like are similar to yours. That's a good thing to look at there. Um, are there other existing benchmarks? There are there are many. Um, we reference many of them in the paper. Um, like there's a lot of prior work I recommend, um, like skimming through it. We we tried to make it like not not it, you know, impenetrably technical. Um, yeah, and you'll you'll definitely find like some of the um some of the popular pre-existing benchmarks are like OmniDocBench, that's a really big one. How do you typically get the ground truth answers at scale for the benchmark when it comes to something difficult to parse like legal documents with handwritten content? Um, and another benchmark, I think they created the ground truth answers by using a large proprietary model. So, yeah, um we go into this in more detail in the paper, but it's a for well, for each of these pages, it's like a a multiple pass flow. Like, there's some human evaluation, there's some language model generation of like a draft here or there, um but there there is always a a human going through and helping evaluate this in like really fine-grained way. Um, just try generating ground truth from a model and like it is in our experience not as reliable as we need to measure like fine-grained accuracy, especially when you start doing really well on content faithfulness, like when you're at 90 plus percent, like if there's a little bit of noise in your ground truth generation, then you're like really no longer able to capture whether you're parsing things correctly. Um, even if it starts deviating from what what the proprietary model generates. Like, it's not that it's not useful, um but for many needs including ours, it's it's not quite sufficient. Cuz we just have like really specific accuracy needs. This is really true for tables, where like um like we said, if you get a single digit wrong, doesn't matter. Like, that's completely wrong. And so, we need to make sure that all those values are correct. Could Could ParseBench be used to create evaluations for custom document parsing solutions? Um, yeah, so like the the Parse Bench you you can run Parse Bench on any of the like providers we've we've benchmarked against, but also against open-source models. Uh we we have like leaderboards for like proprietary and open-source providers at some of these links here. Um So, you can definitely evaluate this against your own your own parsing solution. You just need to like open a PR against our GitHub repo to add the provider. And we have we have instructions at the GitHub. Um will there be leaderboards for tools? Yes. Oh, yep. Yeah, we are working just got to this. Um uh standardized documents. Um If you could clarify more about your question about standardized documents, I'm happy to answer it. I'm not sure exactly what you'd like us to cover here. Um Yeah, okay, so, question here. Um hi Preston, similar question. Understood that this is document parsing, but have you explored extracting data from engineering schematics? So, what were the primary bottlenecks? Yeah, um engineering schematics are very difficult. Um they're like almost entirely different category of document from a lot of others. Um They it's not that they never come up in enterprise workflows, but they like come up in different ones. They certainly come up in construction, right? Um I I think this is almost like an an open question for uh like in the field in general is like the best way to represent these, cuz when you have all these spatial relationships, it's like in fact, probably the most compact way of representing that is the uh diagram itself. Like, what is the actual information that your downstream workflow requires? Um is is that's going to vary, but I think usually you will what you would want access to the source schematic itself. Um I think in the case like that where it's really dense, um you really want visual grounding. You want to be able to attribute it the output to the input. Um yeah, and it may be the case that like extract use cases are the right way to think about it rather than parsing. Um yeah, where like most text-heavy documents you can faithfully represent it as like linear markdown. Um but for schematics, I think this is it's not a given that that's even true. Yeah. Um I think the last question I see here is um is parse bench region-specific? Um I the way that I understand this question is like models have different regions that they operate in. Um and so like did we evaluate these in different in a bunch of different regions? Um for the most part, I think this is like US US SAS. Um like for the the major model providers, um the performance of the different models like can vary a bit depending on whether it's like hosted on Bedrock or something else. Um but for this we we focused on like the like highest performance models at a given given price tier, and this tended to be in in the US just due to like EU rollout delays. Um Oh yeah, and Alicia had asked, "Will there be a leaderboard for tools?" I my understanding of this question is like of the providers, is there a leaderboard of how they perform? Um and yes, like a couple actually definitely exist. We have one on on Hugging Face, I believe, um and another one on Kaggle. Um all of these are are linked to from the like parsebench.ai website. Yeah, like a lot of these link to each other intentionally, so it's easy to find them. Yeah. Um I'm going to pass it over to go to me here. >> Um folks, any more questions? Um if there are no more questions, uh again, this has been uh a great session. Thanks, Preston, and thanks to everyone for such uh I mean, these were like really good questions. Um so, we will share the recording um after the call. Um it'll be in your email. Um so, yeah. Thank you for attending, and we hope to see you soon um in the next one. >> Thank you. >> Thank you very much, and please do please do contribute. Take a look at the at the GitHub. We would love to see your PRs. Thank you. >> Bye, guys.