The Problem With Vision Language Models

If you're using vision language models for document extraction, there's something you need to know. While models like Gemini, GPT-5, and Mistral are incredibly powerful, they are by their very nature generative. They're not extracting text, they're predicting text based on what they see. This can work very well for messy scans or handwriting, but it can also lead to hallucinations. For a lot of use cases, that's fine. But if you need verbatim scans and exact data extraction, the predictive nature of these models is probably not good enough. This is where tools like Docling come in. Docling's standard pipeline includes a series of non-generative AI models that are purpose-built for text extraction, recognizing headers, and understanding table layouts — all while preserving the semantic structure of the document. This standard pipeline isn't necessarily smarter than large vision models. It's just a different tool for the job. You can also run vision models through Docling as well. Docling is completely open source and runs locally, which is why it's our go-to for building fully grounded AI agents. Check out the link below if you'd like to watch how I build and deploy a fully local AI RAG agent.