Advanced Chunking Techniques: Semantic & LLM-Based Chunking (Simply!) Explained

Key Takeaways

Are you ready to learn about how professional rack applications chunk their text? In our last video, we talked about some basic chunking strategies such as splitting large documents up sentence by sentence or section by section. But these chunking methods are missing something. Sentence and section based chunking does not guarantee that each chunk will have an independent meaning. Semantic chunking is designed so that each chunk has a distinct meaning. Here's how it works. First, we take two sentences. Then we calculate how semantically different these sentences are from each other by measuring a similarity score via their vector embeddings. If the similarity score is high, then the sentences are combined together to make a chunk. If the score is too low, then a chunk is formed starting with the new sentence. Alternatively, we could do large language model based chunking. In short, we give the text to a language model and ask it to pick it apart into different sections. This has a ton of advantages, mostly because LLMs have extremely deep understanding of language and can chunk extremely effectively. However, this is a really slow method and could cost us quite a bit of money if we use larger models. Also, how much can we trust the output of an LLM? Lately, there's an even newer chunking method called late chunking that keeps contextual information from one chunk to another. But I'm getting ahead of myself. That will have to wait for a future video. So, stay tuned. [Music]