LangChain Text Splitters & Chunking

Key Takeaways

Welcome back. In this video, we will understand what are document splitters and chunkers. So, we'll get an understanding of document splitters and chunkers. Understand how does a splitter work as well as look at some of the popular built-in document splitters in Langchain. Let's get started. So, what are document splitters and chunkers? Now, Langchain supports various document splitting and chunking mechanisms for typically transforming documents. So the whole idea of splitting documents is you have a large document and you split it into smaller more meaningful or more manageable chunks. Okay. So splitting documents into smaller chunks of paragraphs. This is typically done so that you can fit more relevant chunks into your LLM's context window because often if your large language model has a limited context window which is in most of the cases if you try to fit really large documents and a bunch of them you will get a token limit exceeded error. Okay. And documents can be split based on various methods like relevant sections, character counts, token counts and so on. Now how does a splitter work? At a high level, the text splitter works as follows. You split this text. So let's say you have this complete document text. You split the text into small semantically meaningful chunks. Okay? And then you combine these small chunks into a larger chunk until you reach a certain size based on the character count or the token count. As we have seen in the previous video in document loaders, some of the more complex document loaders, we could also do the chunking where we could define that okay, let every chunk be of 3,000 characters in length. Okay. So once you reach that size, make that chunk its own piece of text and then again start creating a new chunk of text with or without chunk overlaps. We will talk about some of these concepts when we get into the hands-on also. Okay. So how do you split the document into these chunks? The typical splitting strategy can be based on the number of characters. Which means let's say if your chunk size is of 3,000 characters, it will try to fit as many words as possible such that the total length of the characters of all those words is not more than 3,000 characters. It could also be based on the number of tokens or words like let's say you say that the chunk size is 100 tokens which means it will roughly try to fit anywhere between 90 to 100 words. Similarly, you can do semantic or sectionalbased chunking as we saw with the unstructured data loaders where you can chunk the main sections based on the relevant titles or headings of the documents. Okay. And chunk size measurement is typically done in the form of character or token counts. Now what are the popular built-in document splitters in lanch chain? You have the recursive character text splitter which is the most widely used and most popular text splitter which recursively splits text into larger chunks based on well-defined characters and then it tries to keep related pieces of text next to each other and it is lang chain's recommended way to start splitting your text into more manageable chunks. Character text splitter is like a specialized version of recursive character text splitter. It splits text based on just one single userdefined character. It's one of the simpler text splitters. Tik Token helps in splitting text based on tokens based on these pre-trained or fine-tuned LLM tokenizers just like chat GPD 3.5 chat GPD based on GPD version 4 and so on. Spacey is a popular NLP library and you can use spacy also to split text using the tokenizer from this popular spacy library. Sentence transformers is also another popular library. So you can split text based on tokens using trained open large language model tokenizers available from the sentence transformers library and of course as we have seen before in the previous videos also you can use unstructured.io. So the unstructured library allows for various splitting and chunking strategies including splitting text based on characters, key sections, titles and so on. In the next video, we will see how to apply some of these splitting and chunking strategies using some hands-on tech. Thank you and I'll see you in the next