LangChain Document Loaders

Key Takeaways

Welcome back. In this video, we will be looking at document loaders. We will understand what are document loaders, talk about document loaders from unstructured.io, which is a very popular platform to load data from various unstructured data sources and also look at some of the popular document loaders which are integrated into lanchin. So what are document loaders? Now lchain provides a variety of document loaders to load data from a source as a document data type object. So the idea is to take any of these sources which could be PDFs, word documents, PowerPoint presentations, markdown documents and use a document loader to load and extract the text content from it. A document typically has a piece of text content and relevant metadata which could be the file name, the author name, the page number and so on. Langchain has loaders for almost every possible document format and every document loader typically provides a load method for extracting and loading the data as langchain documents from a relevant source from which you are loading the data. Now what about document loaders from unstructured.io? So the unstructured library typically provides open-source components for ingesting and pre-processing a wide variety of document formats which includes PDFs, HTML, Microsoft Word, Excel, PowerPoint and many more. Blankchin has bindings to this unstructured library to access and use its various data loaders. This library has been built by the unstructured.io organization and they have also opensourced it. Now these are some of the popular document loaders which are already available in lang chain. You have a CSV document loader which typically load CSV files into a sequence of document objects. You also have a markdown loader and lang chain leverages unstructured.io to use unstructured markdown loader object which will leverage this library to load data from markdown files. You have a text loader also which is just a simple text loader to load data from text or markdown files. Then you also have JSON where Langchain implements a JSON loader to convert any kind of a JSON or JSON lines data into lang document objects. It uses the jq package to enable extraction of specific fields from the data instead of let's say the full data. We will look at some of these examples in the hands-on video soon. And also PDF. So Lanchin integrates with a variety of PDF passes available. Most of these are open source which includes PI PDF, PI MU PDF which is one of the fastest PDF loaders, PDF minor as well as the unstructure.io PDF loader which also supports OCR image and table extraction. So we will be using this PDF loader even in the future. That's it for this video. I'll catch up with you in the next video where we'll go through a hands-on of document loaders. Thank you.