RAG from scratch: Part 12 (Multi-Representation Indexing)

Our RAG From Scratch video series walks through impt RAG concepts in short / focused videos w/ code. This is the 12th video in our series and focuses on some useful tricks for indexing full documents. Problem: Many RAG approaches focus on splitting documents into chunks and returning some number upon retrieval for the LLM. But chunk size and chunk number can be brittle parameters that many user find difficult to set; both can significantly affect results if they do not contain all context to answer a question. Idea: Proposition indexing (@tomchen0 et al) is a nice paper that uses an LLM to produce document summaries ("propositions") that are optimized for retrieval. We've built on this with two retrievers: (1) multi-vector retriever embeds summaries, but returns full documents to the LLM. (2) parent-doc retriever embeds chunks but returns full documents to the LLM. Idea is to get best of both worlds: use smaller / concise representations (summaries or chunks) to retrieve, but link them to full documents / context for generation. The approach is very general, and can be applied to tables or images: in both cases, index a summary but return the raw table or image for reasoning. This gets around challenges w/ directly embedding tables or images (multi-modal embeddings), using a summary as a representation for text-based similarity search. Code: https://github.com/langchain-ai/rag-from-scratch/blob/main/rag_from_scratch_12_to_14.ipynb References: 1/ Proposition indexing: https://arxiv.org/pdf/2312.06648.pdf 2/ Multi-vector: https://python.langchain.com/docs/modules/data_connection/retrievers/multi_vector 3/ Parent-document: https://python.langchain.com/docs/modules/data_connection/retrievers/parent_document_retriever 4/ Blog applying this to tables: https://blog.langchain.dev/semi-structured-multi-modal-rag/ 5/ Blog applying this to images w/ eval: https://blog.langchain.dev/multi-modal-rag-template/