olmOCR - The Open OCR System

Key Takeaways

okay so I know a lot of people are really interested in OCR and being able to take documents like PDFs and and basically get them into a format that you can put into your llms whether that's for rag whether that's just for dropping it in a context window to be able to use it straight away so this is a challenge that a lot of people face out there and the people at L have released a new model called M OCR so if you don't know who Ln AI is to be really honest I haven't probably made enough videos about them I should have made more videos about them one of the cool things about this lab is that they're not out there trying to make lots of money they're actually trying to make things more open so over the last year or so they've released a bunch of different models that are out there these include the MMO models the Tulu models and one of the really cool things about what they generally do is that they not only release the model and open weights they are actually probably the only group that is really doing proper open source in that they are open sourcing the code that they use to do the training of the model the actual data sets Plus usually a really good paper sort of talking about all the things that they did challenges that they fac that kind of thing so it's really nice to see that they've come along and released this m OCR and they basically talk about that this is an issue that even they've had as well as lots of other people training llms is that you really want to get high quality data for the llm and while you can scrape things off the internet and convert it to plain text and remember plain text is what has to go into the actual training or fine-tuning of a Model A lot of the really high quality data is actually out there in PDFs now the challenge with PDFs is that if they've been kind of printed as a PDF the files have actually been rasterized they're no longer in a text format that we can just easily extract out they're actually pictures now with things like diagrams and other things in there so the idea here is that Ln AI have basically released this model now you need to understand that this model is not something that they've done from scratch they've actually taken the quen 2 VL 7B instruct model and then fine tuned it specifically for this task so I've made multiple videos about the quen models in the past they're really one of the top open weights models that is contributing not only a lot of models in all different sizes from really big to really small to Vision LMS but they're also publishing a lot of papers of how they're doing it people are using their models for doing a variety of different fine tunes out there even the Deep seek R1 32 billion model is actually a fine tune of a quen model so it's not surprising here that L AI have basically decided to use that as their base model and and then come in here and actually fine-tune this so they have a nice blog post where they basically describe what they did and basically how they fine tuned on quarter of a million images in here that were sort of randomly sampled from a big data set and I think that data set includes a large percentage of sort of academic papers but also a variety of different things including handwriting so this model can actually OCR handwriting as well as printed text Etc on top of that the model can actually output in markdown and can handle things like equations and tables and even things like multicolumn documents Etc so I've yet to see any of these models be totally perfect but this one really does seem to be outperforming a lot of the other open-source models that are out there that are doing similar kinds of things like Mara and Miner U Etc so they have a nice sort of interactive tool thing in here where you can sort of look at some things like the math textbook and see okay how are they doing compared to other people historical documents and stuff like that and even handwritten letters that you can see in here the cool thing is if you just want to try this out yourself on a document they've actually put up a demo where you can upload I think it's up to 10 pages it will go through and process for you and if you look at this if we come in here and say look at there the handwriting sort of example we can see that okay it's you know processing pretty quickly and then we can see the output of okay what actually was coming out of this where did it get things right what did it get wrong that kind of thing and while it's perhaps not perfect for doing this kind of thing it certainly is getting really good responses in here we can also see what the actual prompt that they use and stuff like that so if you wanted to do your own prompt you certainly can do that and like I mentioned earlier on one of the cool things about Ellen AI is that they publish everything so if you come over to their GitHub for this you can see that there's a whole thing about installing it and I'll go through this in collab in a second but one of the other things that's really nice in here is that they've actually got fine tuning code that you can sort of Ed for this so you can actually use the same code that they use to actually do the trainings and stuff themselves so if you wanted to extend this for your own special use case you would be able to do this in here so in the blog they've also got like a nice breakdown of actually how they did it talking about the data set you can see here I mentioned early that academic papers was a large percent but also things like brochures legal documents diagrams slides a whole bunch of different stuff has sort of gone into this let's jump into the collab and have a look at how you can get started using this okay jumping into the code you really need to have some bit of a grunt to run this now at the moment I'm using the full resolution version of this there is a GG UF version of this and really nice blog post that I'll put in the description as well of talking about how to run this on Mac OS using LM studio and a quantized version of the model in here but if you do want to run the full version of the model you're going to need a GPU something like an a100 Where've got a pretty decent amount of vram and stuff in there all right so you need to basically install some stuff and utilities to get this going it's actually using SG Lang which is a new inference Library which is interesting I've been looking at that recently so it's interesting to see that they've basically picked up on this as well once you've got everything installed it's pretty easy to get it going it's relying on the Transformers library and you're going to bring in the setup for the quen 2 VL model that's the 7B instruct Etc you can see if you remember I've talked about this in the past that in these cases you've got the actual model and then you've got a processor and the processor is basically a tokenizer for both the words in this case decoding those words but also for the images in there so autoprocessor basically is just the equivalent of sort of tokenizer that we use when we're just using a standard llm model in here the model is not hugely big and stuff like that so that's something you can get started quite easily you're going to need your PDFs obviously so you just want to basically bring in the PDF and then you basically need a way to render that PDF to an image so in this case I'm basically taking in that PDF I pass in what page number I want and this is going to render it to a base 64 image in here so I've added in some code to plot that out so that we can actually just see what the plot of this actually is like and if we come up here and we change the page here and change that you'll see that sure enough we get a different page from the paper coming out as we go through this all right next up we've got this whole sort of system of where they're basically working out this sort of anchor text and stuff like that and then combining this in with their prompt you're running it through just like a normal sort of Transformers thing where we're passing in a message that message just happens to also include this base 64 image going through there you want to make sure that you set the number of tokens out I think by default their example only have 50 or 100 tokens or something like that you will then basically get a cut off sort of Jason string so you'll see there there's no closing of the brackets so you want to make sure that you actually get it to produce the full amount so that it does actually close off the Json for you and then you can basically just import this in and if we bring that string in and convert it to Json and look at it we can just sort of print out the key value pairs you can see that okay it's going to basically have a thing about what languages it is is there a rotation going on is there a table is there a diagram and then the natural text and this is basically the text for it so you can see this is the title of the paper we've got the authors here and then we got the abstract going on here and it will sort of output it quite long and it will output it in there now you can see that's fitting with what we're seeing in here we've even got the introduction so it's done it so you can see that it's worked out that this text should come before this text right in we've got the abstract up there and then we've got the introduction in there all right so what I've done is just put it all together into a couple functions and stuff so that you could just run a full paper through this now I will say that if you want to do something sort of serious with this and not just have a play with it you probably don't want to use what I've got here because this is basically just processing the pages sequentially what you really want to do is if you're doing any sort of high volume production you're better to use batch mode and run things through in batches as you're going through this now they actually have a toolkit for this I'll just sort of come I've put this down at the end here unfortunately this is not working in collab easily at the moment so I've basically just put it together in some functions where we can sort of set it up to go through here I've added in a little function to basically count the number of pages and then we can just run it through one page at a time in this case so like I said this is not the most optimized way of doing it but it is actually working and getting it through from that we can then basically join these out and you can see sure enough we've now got our paper being converted into this natural text that we've got out show you'll see that we've got things like markdown as we go through it we've got tables so you can see here the tables that it's basically putting together which are quite nice it doesn't have huge support for things like diagrams so you're not going to get like really nice descriptions of diagrams or something like that but this is certainly a good step forward for being able to process all this stuff locally yourself and get it into a format that you can actually start to basically save this into file and bring it into llm so if you're sitting on a bunch of PDF files that you want to convert that you want to use and you want to convert for this and you don't want to send them to a commercial Cloud cuz things like Gemini Flash and stuff can also do this quite nicely the only challenge that you've got there is that your data is going off Prem so if you want sort of like an on Prem solution this is something that you can basically use set up very quickly and get started in here anyway have a play with it I've been trying it out both on the ones that I've got here and also on some handwriting documents so far it's been very impressive that like this is quite easy and good and having something that can run on Prem with a GPU or using the local version with something like LM Studio to get this running on your PC is a really nice alternative compared to having to send everything out to the cloud for doing this kind of stuff I do think there's still a lot of opportunities for using things like flash and stuff like that if you don't really care about the privacy of the documents and stuff stuff that's going to be really both cheap and very quick and economical for how you can do that as well all right as always I'd love to hear in the comments what you're sort of interested in using this for what the big challenges that you're having in relation to this kind of thing and if you've got any questions or anything please put them you know below as well if you found the video useful please click and subscribe and I will talk to you in the next video bye for now