Classic RAG Limitations Explained

Key Takeaways

Welcome back. In this video, we are going to cover what is rag and where it breaks. By the end of this video, you will have a clear picture of how classic rack works and the exact point where it starts to fall apart. Let's start with the basics. What exactly is rag? Rack stands for retrieval augmented generation. It works in three steps. First, retriever. Find the most relevant content from your document based on the user's query. Second, augment. take that retrieved content and add it to LLM prompt as context. And third is generation. The LLM produces a grounded accurate answer using that context. That's the core idea. Simple and powerful. But as we will see, there are specific situations where this approach breaks down. Now let's look at how this actually works end to end. Your source documents, whether PDFs, web pages, or any text files, goes through a chunker that splits them into smaller pieces. Each chunk is passed through an embedding model that converts text into vectors. Those vectors are stored in vector store. When a user sends a query, it goes through the same embedding model. The retriever then searches the vector store for the most similar chunks. Those chunks are passed into LLM context window along with the original question and the LLM generates the final answer. This is the classic rag pipeline. Keep this flow in mind as we walk through where things go wrong. Before we critique rag, let's be fair about where it genuinely works well. Factual lookups from a fixed document set works great. If someone askked what is your refund policy and that answer sits cleanly in one document, Rag will find it. Single hop question answer over internal documents works well too. Questions like what does the contract say about termination have self-contained answer that rag handles reliably and rag does meaningfully reduce hallucination. The LLM now has real retrieved content to work from instead of guessing from training data alone. Rag is not broken. It is solid system for specific class of problems. The question is what happens when your queries get more complex? Here is the first failure mode multihop queries. Consider this question. Which researchers influence the methodology used in this paper and what institution are they affiliated with? What rack does is it retrieves the chunk closest to that query text maybe the abstract and stops there. It misses the citation chain entirely. What is actually needed is a chain of reasoning. Paper cites author Affiliated with institution X who also collaborated with author B. That is three hop of reasoning. Rag has no mechanism to chain reasoning across multiple documents or entities. It makes one retrieval jump not three. That is the core limitation here. The second failure mode is one of the most important. Rag has no awareness of relationship between entities. Take this query. How are company A and company B connected? What rag does? It searches for chunks containing both names and written co-occurrence. But co-occurrence is not connection. What is actually needed is a traversal. Company A acquired subsidiary X which was founded by person Y who sits on the board of company B. That chain of relationship does not live in any single chunk. It exists in the structure between documents in the edges between entities. Rag has no way to see that structure. It treats every chunk as an isolated island. The third failure mode is chunk loss and ambiguity. And this one is very common in practice. The chunking problem. The answer often sits across two chunks. The chunker splits at a fixed token count and neither chunk alone is useful. You lose the answer at the boundary. Now the ambiguity problem. Tell me about the Apple announcement. Which apple? Which announcement? Which air? Vector similarity retrieves whatever is the closest to that query, not what users actually meant. Rag has no disambiguation layer. It matches text not intent. These are not rare edge cases. They happen constantly in real world rack deployments. That wraps up video one. We now have a clear picture of what classic rack is, how it flows end to end, and three specific places where it breaks. Multihop queries, no relationship awareness and chunk loss with ambiguity. In the next video, we will see how graph rack is designed to solve all three. See you there.