Developing Applications with LangChain | Master LLMs, RAG, and Agentic Workflows

Key Takeaways

what's up everybody Adell here today's video is a collection of course videos from our track on developing applications with Lang chain in these videos you'll learn all about retrieval augmented generation with langen chin including how to use rag graph rag as well as other techniques as well as Building agentic Systems using L chain and more these videos will only cover the video portion of the courses so if you're looking to practice your skills right in the browser make sure to check out the link in the description below with that happy learning hello I'm Jonathan benan an AI engineer and active contributor to linkchain Welcome to the course we now live in a world where we're spoiled for Choice when it comes to choosing large language models or llms for developing AI powered applications however these llms differ based on their model architecture training data intended use cases and prompting strategies a additionally these models often have to interact with other systems to retrieve data or monitor performance which adds another layer of complexity Lang chain is an open- Source framework that helps developers connect llms data sources and other functionality under a single unified syntax with Lang chain developers can create scalable modular llm applications with greater ease this course will cover Lang chain and python but libraries also exist for JavaScript Lang chain encompasses an entire ecosystem of tools but in this course will focus on the core components of the linkchain library llms including open source and proprietary models prompts chains agents and document retrievers throughout the course you'll see how these L chain components can enable some truly awesome llm applications let's first discuss llms hugging face is a huge repository of Open Source data sets tools and most importantly for US models accessing llms hosted on hugging face is free but using them in Lan chain requires creating a hugging face API key to create One log in or create a hugging face account and navigate to the URL shown under settings here click on new token and copy your key data Camp won't store the API Keys you'll need in this course so they can be directly copied into the exercises now we have our key let's use Lang chain to use a model from hugging face and compare it to an open AI model Langan chain has open Ai and hugging face classes for interacting with their respective apis once imported we Define the llm with the model name and API key from hugging face we're using using the Falcon 7 billion parameter instruction optimized model we'll Define an unfinished sentence and use both models to predict the next words finally let's print the result to see the output compare the two different approaches despite using completely different models from different apis Lang chain unifies them both into a consistent modular workflow to summarize Lang chain is a Fant fantastic tool for developing and orchestrating natural language systems in the real world of development for production it unlocks the ability for intelligent conversations with documents more opportunities for task Automation and different ways to analyze text Data linkchain makes implementing AI more intuitive and gives us greater control over the entire workflow note that we will use a specific Lan chain version for this course if you wish to use a newer version on your system be sure to check the Lan chain docs for any changes welcome back let's use Lan chain to start implementing prompting strategies for chatbots thousands of llms are available in linkchain via the hugging face Hub API to find language models specifically optimized for chat search the models section of hugging phas and filter on question answering then take note of the model name so it can be referenced in the code new models are constantly being released on hugging face many are also fine-tuned on domain specific data sets so they are better at capturing the Nuance of a particular region culture or task so it's worthwhile to spend time searching hugging phase for the most appropriate model once we've selected a model we can begin prompting it by utilizing prompt templates prompt templates act as reusable recipes for generating prompts from user inputs in a flexible and modular way templates can include instructions examples or any additional context that might help the model complete the task prompt templates are created using link Chain's prompt template class we start by creating a template string which is structured to prompt the AI to answer a question the curly braces indicate that we'll use Dynamic insertion to insert a variable into the string later in the code to convert this string into a prompt template we pass it to prompt template specifying any variables representing inputs using the input variables argument to show how a variable will be inserted let's call the invoke method on the prompt template and pass it a dictionary to map values to input variables we see in the input how the question placeholder has been replaced let's integrate this with an llm we start by choosing a question answering llm from hugging phase to integrate The Prompt template and model we use Lang chain expression language or L cell using a pipe creates a chain which in linkchain are used to connect a series of calls to different components into a sequence to pass an input to this chain we call the invoke method again with the same dictionary as before this passes the question into the prompt template then passes The Prompt template into the model for a response chat models have a different prompt template class chat prompt template this allows us to specify a series of messages to pass to the chat model this is structured as a list of tupal where each tupal contains a role and message pair this list is then passed to the from messages method to create the template in this example we can see three roles being used system human and AI the system role is used to define the Model Behavior the human role is used for providing inputs and the AI role is used for defining outputs which is often used to provide additional examples for the model the chat openai class is used to access open ai's chat models when instantiating the model make sure to provide an open AI API key we create our chain again using an L cell pipe Define a user input and invoke the chain as before the output is pretty long but we could tweak our prompt templates to change this welcome back in this video we'll take our llm prompting skills up a level and master fuse shot prompting so far we've used prompt template and chat prompt template to create reusable templates for different promps inputs these classes are great for handling prompts containing a single example but they don't scale well if we need to integrate many examples from a data set the fot prompt template class allows us to convert data sets like these into prompt templates to provide more context to the model let's say we have a list of dictionaries containing questions and answers if we have another data structure like a panda's data frame there's usually a simple transformation to get to this point like the two dict method in this case before we dive in with creating the fuse shot prompt template we need to decide how we want to structure the examples for the model we create the prompt template using the prompt template class we've used before to specify how the questions and answers should be formatted invoking this template with an example question and answer we can see the question prefix was added and a new line was inserted now to put everything together fuse shot prompt template takes the examples list of dictionaries we created and the template for formatting the examples additionally we can provide a suffix which is used to format the user input and specify what variable the user input will be assigned to let's see our prompt in action we invoke The Prompt template with an example user input and extract the text from the resulting prompt the prompt correctly appended question to the user input displaying it underneath the example questions and answers now let's test this prompt template is actually functional in an llm chain we instantiate our model and chain The Prompt template and model together using the pipe operator from lell the model response can be extracted from the response object via the content attribute which shows the model was able to use the context provided in our fuse shot prompt let's take our L cell chains to the next level with sequential chains some problems can only be solved sequentially consider a chatbot used to create a travel itinerary we need to tell the chatbot or destination receive suggestions on what to see on our trip and tell the model which activities to select to compile the itinerary this is a sequential problem as it requires more than one user input one to specify the the destination and another to select the activities let's code this out in sequential chains the output from one chain becomes the input to another we'll create two prompt templates one to generate suggestions for activities from the input destination and another to create an itinerary for one day of activities from the model's top three suggestions we Define our model and begin our sequential chain we start by by defining a dictionary that passes our destination prompt template to the llm and parses the output to a string all using L cell's pipe this gets assigned to the activities key which is important as this is the input variable to the second prompt template we pipe the first chain into the second prompt template then into the llm and again pars to a string we also wrap the sequential chain in parentheses so we can split this code across multiple lines to summarize the destination prompt is passed to the llm to generate the activity suggestions and the output is parsed to a string and assigned to activities this is passed to the second activities prompt which is passed to the llm to generate the itinerary which is parsed as a string let's invoke the chain passing Rome as our input destination the model considered that we only had one day to explore and wo in its top suggestions of the Coliseum and Vatican City now that we're confident with llms it's time to move on to a hot area of AI application development agents in linkchain agents use language models to determine actions agents often use tools which are functions called by the agent to interact with the system these tools tools can be highlevel utilities to transform inputs or they can be task specific agents can even use chains and other agents as Tools in this video we'll discuss a type of agent called react agents react stands for reasoning and acting and this is exactly how the agent operates it prompts the model using a repeated Loop of thinking acting and observing if we were to ask a react agent that had access to a weather tool what is the weather like in Kingston Jamaica it would start by thinking about the task and which tool to call call that tool using the information and observe the results from the tool call to implement agents we'll be using Lane graph which is a branch of the linkchain ecosystem specifically for Designing agentic systems or systems including agents like Lang Chain's core Library it is built to provide a unified tool agnostic syntax we'll be using the following version in this course we'll create a react agent that can solve math problems something most llms struggle with we import create react agent from Lane graph and the load tools function we initialize our llm and load the llm math tool using the load tools function to create the agent we pass the llm and tools to create react agent just like s agents can be executed with the invoke method here we passed the chat model a message to find the square root of 101 which isn't a whole number let's see how the agent approaches the problem there's a lot of metadata in the output so we've trimmed it for brevity we can see that executing the agent resulted in a series of messages the first is our prompt defining the problem the second is created by the model to identify the tool to use and to convert our quy into mathematical format the third is the result of the tool call and the final message is the model's response after observing the Tool's answer which it decided to round to two decimal places if we just want the final response we can subset the final message and extract its content with the content attribute now let's begin implementing agents now that we've created our first agent let's take a closer look at tools so we can design our own tools in linkchain must be formatted in a specific way to be compatible with agents they must have a name accessible via the name attribute a description which is used by the llm to determine when to call the tool in this case if the llm interprets the task as being a math problem it will likely call this tool based on its description finally the return Direct parameter indicates whether the agent should stop after invoking this tool which it won't in this case understanding this required format will help us to understand how to create our own tools let's say we want to define a python function to generate a financial report for a company it takes three arguments the company name revenue and expenses and outputs a string containing the net income we make the use of this function clear in the doc string defined using triple quotes here's what the report looks like let's convert this function into a tool our agent can call to do this we import the tool decorator and add it before the function definition don't worry if you're not familiar with python decorators the at tool modifies the function so it's in the correct format to be used by a tool like with the built-in tool we were looking at we can now examine the various attributes of our tool these include its name which is the function Name by default its description which is the function doc string and return direct which is set to false by default we can also print the tools arguments which lay out the argument names and expected data types let's put our tool into action we'll again use a react agent combining the chat llm with a list of tools to use containing our new custom tool we invoke the agent with an input containing the required information a company name revenue and expenses the response from the agent starts with our input then determines that the financial report tool should be called which returns a tool message containing the output from our function and finally the output is passed to the llm which responds to us let's zoom in on this final message here's the final output from the llm notice anything here's the output from the tool based on the function we defined notice that there's slight formatting differences between the two the llm received the tool output and put its own slight spin on it which you may need to watch out for in this chapter we'll discuss retrieval augmented generation or rag pre-trained language models don't have access to to external data sources their understanding comes purely from their training data this means that if we require our model to have knowledge that goes beyond its training data which could be company data or knowledge of more recent world events we need a way of integrating that data in rag a user query is embedded and used to retrieve the most relevant documents from the database then these documents are added to the model's prompt so that the model has extra context to performance response there are three primary steps to rag development in Lan chain the first is loading the documents into L chain with document loaders next is splitting the documents into chunks chunks are units of information that we can index and process individually the last step is encoding and storing the chunks for retrieval which could utilize a vector database if that meets the needs of the use case we'll discuss all of these steps throughout the next chapter but for now we'll start with document loaders Lan chain document loaders are classes designed to load and configure documents for Integrations with AI systems linkchain provides document loader classes for common file types such as CSV and PDFs there are also additional loaders provided by Third parties for managing unique document formats including Amazon S3 files Jupiter notebooks audio transcripts and and many more in this video we will practice loading data from three common formats PDFs csvs and HTML linkchain has excellent documentation on all of its document loaders and there's a lot of overlap in syntax so explore at your leisure there are a few different types of PDF loaders in linkchain and there's documentation available online for each in this video we'll use the pi PDF loader we instantiate the PIP PDF loader class passing in the path to the PDF file we're loading finally we'll use the load method to load the document into memory and assign the resulting object to the data variable we can then check the output to confirm we have loaded it note that this document loader requires installation of the P PDF package as a dependency when loading csvs the syntax is very similar but instead we use the CSV loader class we're seeing a pattern forming finally we can load HTML files using the unstructured HTML loader class we can access the documents contents again with subsetting and extract the documents metadata with the metadata attribute now that we've loaded documents from different sources let's learn how to parse the information document splitting splits the loaded document into smaller Parts which are also called chunks chunking is particularly useful for breaking up long documents that they fit within an llm context window let's examine the introduction from an academic paper which is saved as a PDF one naive splitting option would be to separate the document by line this would be simple to implement but because sentences are often split over multiple lines and because those lines are processed separately key context might be lost to counteract lost context during chunk splitting a chunk overlap is often implemented we've selected two chunks and a chunk overlap shown in green having this extra overlap present in both chunks helps retain context if a model shows signs of losing context and misunderstanding information when answering from external sources we may need to increase this chunk overlap there isn't one document splitting strategy that works for all situations we should experiment with multiple methods and see which one strikes the right balance between retaining context and managing chunk size we will compare two document splitting methods character text splitter and recursive character text splitter optimizing this document splitting is an active area of research so keep an eye out for new developments as an example let's split this quote by Elbert hubard which contains 103 characters into chunks we'll compare how the two methods perform on this quote with a chunk size of 24 characters and a small chunk overlap of three let's start with character text splitter this method splits based on a separator first then evaluates chunk size and chunk overlap to check if it's satisfied we call character text spitter passing the separator to split on along with the chunk size and chunk overlap applying the splitter to the quote with the split text method and printing the output we can see that we have a problem each of these chunks contains more characters than our specified chunk size character text splitter splits on the separator in an attempt to make chunks smaller than chunk size but in this case splitting on the separator was unable to return chunks below our chunk size let's take a look at a more robust splitting method recursive character text splitter takes a list of separators to split on and it works through the list from left to right splitting the document using each separator in turn and seeing if these chunks can be combined while remaining under chunk size let's split the quote using the same chunk size and chunk overlap notice how the length of each chunk varies the class split by paragraph s first and found the chunk size was too big likewise for sentences it got to the third separator splitting words using the space separator and found that words could be combined into chunks while remaining under the chunk size character limit however some of these chunks are too small to contain meaningful context but this recursive implementation may work better on larger documents we can also split other file formats like HTML recall that we can load HTML using unstructured HTML loader defining the splitter is the same but for splitting documents we use the split documents method instead of split text to perform the split welcome back now that we've covered document loading and splitting we'll round out the rag workflow with learning about storing and retrieving this information using Vector databases we've now loaded documents and split them into chunks using an appropriate chunk size and chunk overlap all that's left is to store them for retrieval we'll be using a vector database to store our documents and make them available for retrieval this requires embedding our text documents to create vectors that capture the semantic meaning of the text then a user query can be embedded to retrieve the most similar documents from the database and insert them into the model prompt there are many Vector databases available in Lang chain when making the decision on which solution to use consider whether an open source solution is required which may be the case and high customizability is required also consider whether the data can be stored on off premise on thirdparty services not all cases will permit this the amount of storage and latency of retrieving results is also a key consideration sometimes a lightweight in-memory database will be sufficient but others will require something more powerful in this course we will use chroma DB because it is lightweight and quick to set up we'll be storing documents containing guidelines for a company's marketing copy there's two guidelines one around brand capitalization and another on how to refer to users now that we've parsed the data it's time to embed it we'll use an embedding model from open aai by instantiating the Open aai embeddings Class passing in our openai API key to create a chroma database from a set of documents call the from documents method on the chroma class passing the documents and embedding function to use we'd like to persist this database to disk for future use so provide a PA to the persist directory argument finally to integrate the data datase with other linkchain components we need to convert it into a retriever with the as retriever method here we specify that we want to perform a similarity search and return the top two most similar documents for each user query so the model knows what to do will construct a prompt template which starts with the instruction to review and fix the copy provided insert the retrieved guidelines and copy to review and an indication that the model should should follow with the fixed version to chain together our retriever prompt template and llm we use l cell in a similar way as before using pipes to connect the three components the only difference is that we create a dictionary that assigns the retrieved documents to guidelines and assigns the copy to review to the runnable pass through function which acts as a placeholder to insert our input when we evoke the chain printing the result we can see the model fix the two guideline breaches congratulations on completing the course let's wrap up by reviewing what you've learned we started by talking about the core components of Lan chain models prompts chains agents and document retrievers we used open source models from hugging phase as well as proprietary models from open AI we learned how to combine models prompts and agents using lell chains finally fin we learned how to integrate external data into llms overcoming the limitations of their training data as you continue your Lang chain journey into AI application development the Lang chain Hub will be a fantastic resource along the way the Hub contains a catalog of prompts for a whole range of different tasks you can search use and add to this massive catalog as you progress Lang chain templates are also worth investigating there are sets of code blocks that are ready to use out of the packet and they cover many of the most common use cases they may require minor modifications or additions but they're often the best place to start the core link chain package that you've seen in this course is only a piece of the full Lan chain ecosystem which also includes Lang Smith Lang serve and Lane graph as you've also seen in this course Lang Smith is used for troubleshooting and evaluating llm applications and laying serve is used for deploying these applications to production while laying graph as you know is used for multi-agent knowledge graphs continue to explore these tools as the ecosystem evolves hello everyone I am Mary Nova a machine learning engineer founder at break into data and a content creator welcome to this course on rag with Lang chain large language models or llms have become integrated in many of the systems and applications that we interact with in our day-to-day lives one of their key limitations is that their knowledge base is constrained by what was included in their training data in this course we'll explore how retrieval augmented generation or rag allows us to overcome this by integrating external data sources into llm applications this works by embedding user queries to retrieve relevant documents and incorporating them into the models prompt rag provides extra context for more informed llm responses this method is commonly used to provide more relevant answers to users based on company's external proprietor data to enable a rag workflow we need to set up our data sources for retrieval which starts with loading the documents to build up the knowledge base splitting them into chunks to be processed in creating numerical representations from text called embeddings these embeddings or vectors are stored in a vector database for future retrieval in this video we'll start with loading documents with Lang chain Lang chain document loaders facilitate the integration of documents into AI systems these loaders handle various file types including standard formats like CSV and PDFs as well as specialized formats supported by thirdparty providers such as Amazon S3 files Jer notebooks and audio transcripts in this video we'll cover how to use document loaders to import CSV PDF and HTML files into Lang chain let's let's start by loading CSV files using CSV loader here's a quick example we instantiate the CSV loader class passing it the path to the CSV file we want to load assigning the result to CSV loader to load these documents into memory we call the do load method on the document loader each document has page content and metadata attributes to access the respective data next we'll use the pi PDF loader to load PDF files PDFs are a commonly used document format that can store text and images like CSV loader this class takes a file path to create the document loader and has a do load method to load the document into memory we're starting to see a pattern here lastly we'll look at loading HTML files using unstructured HTML loader HTML files can be tricky due to their complex and highly nested structure but this Lang chain class simplifies the process loading the documents into memory and viewing the page content and metadata of the first document we can see that the HTML tags used to structure the page have been removed leaving only the plain text welcome back previously we learned how to load our documents for retrieval next we need to split documents into chunks that can be quickly retrieved and integrated into the model prompt this requires embedding these chunks so they can be retrieved based on their semantic similarity and storing them in a vector store let's cover these final three starting with splitting sometimes called chunking ideally documents are split split into chunks that contain sufficient context to be useful to the llm however larger doesn't always mean better if the chunks are huge retrieval will be slow and the llm May struggle to extract the most relevant context from the chunk to respond to the query the chunk size parameter is used to control this balance another parameter chunk overlap is used to capture important information that may be lost around the boundaries between chunks let's try our first splitting method character text Splitter on this text string first instantiate a splitter using the class we specify the separator to split on in our case it will split the text with each new paragraph with chunk size below 100 and 10 overlapping characters to apply the splitter call the dot split text method on the text let's view the chunks and their lens which we obtain using a list comprehension keep in mind that character text splitter frequently creates chunks that lack sufficient context to be useful in retrieval like the first chunk here the method was also unable to create chunks that were all below the chunk size by splitting by paragraph We can improve on this with recursive character text splitter it takes a list of separators and recursively splits using each one attempting to create chunks below chunk size for example the first separator creates chunks that exceed 100 characters will be split further using the next separator recursive character text splitter often preserves more contexts which will result in more coherent responses from our rag application extending splitting strings to splitting documents requires just one change swapping the Dos split text method toplit documents each document has page content and metadata attributes for extracting the respective information calculating the number of characters in each Chunk we can see that we were able to stay under the chunk size now we've split documents into chunks let's embed and store them for retrieval remember that embeddings are numerical representations of text embedding models aim to capture the meaning of the text and these numbers map the text position in a high-dimensional or vector space Vector stores are databases specifically designed to store and retrieve this high-dimensional Vector data when documents are embedded and stored similar documents are located closer together in the vector space when the rag application receives a user input it will be embedded and used to query the database returning the most similar documents we'll use an embedding model from open Ai and store the vectors in a chroma Vector database let's start by initializing the model then to embed and store the chunks in one operation we call the Dot from documents method on the chroma class passing the chunks and model note that if we were embedding string chunks we'd use the Dot from text method instead great work so far now we have our external data stored and ready for retrieval it's time to create the retrieval chain to integrated it with an llm linkchain expression language or LC L is a declarative Syntax for defining chains suited from prototypes to production it is particularly relevant to Rack systems because it creates modular reusable pipelines that can combine retrieval and generation components together the retrieval chain will take a question input insert it into the chain using runable pass through and assign it to question runnable pass through allows in puts to be inserted into chain unchanged we retrieve the relevant documents from the vector store and assign to context integrate both of these into a prompt template pass the prompt to the model to generate an output and parse the output into our favored format such as a string before building the chain we need to create three components a retriever which is derived from our vector store a prompt template for combining the user question and retrieved context and a model to generate the response previously we created a vector store from embedding document chunks and using the Dot from documents method to convert this into a retriever suitable for our chain we call the S retriever method on Vector store the arguments dictate what sort of search should be conducted during the retrieval the similarity search in this case and how many chunks to retrieve when queried K which is two to construct a prompt template for the model we'll use chat prompt template inside we instruct the model to use the context provided to answer the question we use curly brackets to indicate placeholders for the two input variables context and question we'll Define an open AI chat model to generate the responses now let's explore how to build a chain using LC and rable plus pass through we start by opening parentheses so we can Define our chain over multiple lines then create a dictionary that takes the input from runnable pass through assigns it to question and uses it to query and retrieve chunks from the retriever which are assigned to context this might look complex but run will pass through is essentially a placeholder in our chain that allows us to pass data through without modifying it the retrieved contexts and user question are then passed into the prompt using the LCL pipe then into the llm finally a string output parser is used to parse the model output as a string given that our Vector store contains chunks from rag academic paper let's invoke the chain to request the paper paper's findings we can see that context from the paper was successfully retrieved and used by the model to respond to the question at this stage we might wonder how we can evaluate and improve our model's Performance Based on its response stick around to find out in this chapter we'll dive deeper into building more sophisticated rag architectures with L chain starting with methods for loading and splitting code files we've previously looked at a few common file formats including PDFs csvs and HTML files let's extend this a little further to Python and markdown files which are both common in software engineering and data science projects let's start with markdown files markdown is a lightweight markup language for creating formatted documents and it's often the tool of choice for writing code documentation when Rend markdown can show links images code blocks and much more the unstructured markdown loader class can be used to load markdown files the same way as other file formats we've looked at before by instantiating the class on the file path and using the do load method to load it into memory we could integrate these documents into aack application to read code documentation and make recommendations now now on to python files imagine we have a code base and would like to have a way to talk with it and ask a questions we could achieve this by integrating python files into a rag application the python loader class and the do load method can be used to load these files into memory the resulting documents have page content and metadata attributes for accessing the documents details remember parsing python files can be tricky because it has its own syntax with Imports classes functions and much more that needs to be preserved during chunking let's see how we can address that let's start splitting this python file with our current best tool for document splitting recursive character text splitter we set a chunk size and chunk overlap to control the number of characters in each chunk splitting the documents withit documents method we can print the content of each chunk here's the result notice that the split between chunks 2 and three splits the anthropic class and because chunks are processed separately key context has been lost our current strategy is naive because it doesn't even consider structures like classes and functions let's change this let's split our loaded python file using recursive character text splitter again but this time let's use the do from language method this method has a language argument which refers to coding languages that we can set to language. Python and the rest of the arguments stay the same this will modify the default separators list from the hierarchy of paragraphs sentences and words to try splitting on classes and function definitions before moving on to the standard separators now let's calls spit documents to perform the split and view the first three chunks again as we can see the splitter was able to split on class definitions so all of that context is kept together note that this approach isn't final and depending on the size of the classes and functions relative to the chunk size we may get a differing result now let's level up our document splitting our current approach for splitting documents with character text Splitters has a few limitations these splits are effectively naive because they are ex executed without considering the context of the surrounding text this means related information will potentially be stored in process separately which will lower the quality of our rag application splits are also made using characters rather than tokens recall that the language models we're using break takes text into tokens or smaller units of text for processing if we split documents using characters rather than tokens we risk retrieving chunks and creating a retrieval prompt that exceeds the maximum amount of text the model can process at once also called the model context window we'll introduce methods to make our splitter more aware of the documents context and enable splitting with tokens when we use a character text splitter with a chunk size and chunk overlap we get chunks containing groups of characters that satisfy the chunking parameters when we're split on tokens the chunk size and chunk overlap refer to the number of tokens in the chunk rather than characters so a chunk size of five means so we can have a maximum of five tokens in the chunk here there are five tokens in the first chunk each colored differently four in the second chunk and a two chunk overlap the token teex splitter can be used to perform token SP splitting it requires the name of the encoding to use which is the encoding used by the large language model and can can be retrieved using the tick toen do encoding for model method and extracting the name with aame attribute remember the chunk size and chunk overlap now represents tokens rather than characters we'll use the do spit text method to split the example string and view the chunks this looks good but we're able to keep the chunk size of 10 tokens let's check we can Loop through our chunks encoding them into tokens with encoding do encode and calculating the number of tokens with Len the first chunk contains 10 tokens the second chunk has six and there's an overlap of two tokens fleece was so everything worked as expected let's now learn about a splitting strategy to split in a more context aware way semantic splitting take this block of text for example a character or token text splitter will split naively which result in Lost context a semantic splitter will detect shifts in semantic meaning and perform the splits in those locations in this example when the discussion shifts from rack to dogs to perform semantic splitting we'll need an embedding model to generate text embedding to determine the shift in topic we'll use a model from open AI we instantiate the semantic splitting class passing the embedding model we pass two additional parameters breakpoint threshold type which sets the metric at which embeddings are compared and breakpoint threshold amount which sets the metrics Threshold at which to perform the split like other Splitters we use the do split documents method to apply the split in this case to an academic paper the semantic splitter reached the threshold of 0.8 and performed the splits for the first chunk splitting after the first two sentences of the abstract now that we've upgraded our document loading and splitting let's see how the document retrieval process can be optimized so far our document retrieval has consisted of a vector database containing embedded documents the input to the rag app application is then used to query the vectors using a distance metric to determine which vectors are closest and therefore most similar and relevant this type of retrieval is known as dense retrieval dense retrieval methods encode the entire chunk is a single Vector that is said to be dense that is most of its component values are non zero dense retrieval excels at capturing semantic meaning but the embeddings can be computationally intensive to create and query and may struggle with capturing rare words or highly specific technical terms there's also sparse retrieval which is a method of finding information by matching specific keywords or terms in a query with those in documents the resulting vectors contain many zeros with only a few non-zero terms which is why they're said to be sparse sparse retrieval allows for precise retrieval matching on exact word words the resulting vectors are also more explainable due to the alignment with specific terms and rare words are better represented in the embeddings the trade-off here is that sparse retrieval methods are less generalizable as they aren't extracting the semantic meaning from the text TF IDF and bm25 are the two popular methods for encoding texts as sparse vectors tfidf or term frequency inverse document frequency creates a sparse Vector that measures a terms frequency in a document and rarity in other documents this helps in identifying words that best represent the document's unique content bm25 or best matching 25 is an improvement on tfidf that prevents high frequency words from being overemphasized in the encoding let's try out bm25 for rag retrieval the BM 25 retriever class can be used to create a retriever from documents or text just like the retrievers we have already used let's start with a small example of three statements about python we can use the do from text method to create the retriever from these strings the K value sets the number of items returned by the retriever when invoked invoking the Retriever with an input and returning the page content of the most relevant result according to bm25 reveals that the statement about when python was created was correctly returned looking at all three statements again we can see that bm25 returned the statement with similar terms to the input that were also unique to the other statements now that we've tested out the bm25 retriever let's integrate it into rag we'll create a rag system to integrate a data cam block post on rag with an llm the first step is the same as before but using the Dot from documents method as we're dealing with document chunks and not strings this time then we use the same LCL syntax as a standard dance retrieval rack to integrate the Retriever with a prompt template and llm remember that runable pass through allows us to insert the input and change into the chain finally we can invoke the chain on the input and return the results welcome back because our rag architecture is made up of several processes there are few places where performance can be measured we can evaluate the retrieval process to check if the retriev documents are relevant to the query the generation process to see if the LM hallucinated or misinterpreted The Prompt or the final output to measure the performance of the whole system let's start with the final output we can use llms to measure the correctness of the final output by comparing it to a reference answer we'll assign the query models answer and reference answer to compare with the following variables to perform string evaluation we need to define a prompt template and large language model to use for evaluation The Prompt template instructs the model to compare the strings and evaluate the model output for correctness returning correct or incorrect the model temperature is also set to zero to minimize variability we initialize L chain string evaluator from Lang Smith which is Lang Chain's platform for evaluating llm applications this evaluator first takes QA which sets the evaluator to assess correctness and also the llm and prompt template to use we then call the dot evaluate strings method on the model prediction reference answer and input query to perform the evaluation the score of zero indicates that predicted response was incorrect when compared to the reference answer and we can see here that the model response was deemed incorrect which makes sense on reviewing it again now let's explore further with ragas framework ragas was designed to evaluate both the retrieval and generation components of ARA application we'll cover one metric for each component faithfulness and context Precision faithfulness assesses whether the generated output represents the retrieved documents well it is calculated using llms to assess the ratio of faithful claims that can be derived from the context to the total number of claims because faithfulness is a proportion it is normalized to between zero and one where a higher score indicates greater faithfulness ragas integrates nicely with Leng chain and the first step involves defining the models for the evaluator to use one for generation and another for embeddings next we Define an evaluation chain passing it the faithfulness metric from Ras in the two models we defined to evaluate a model's response we instantiate the chain passing it a dictionary with question answer and context Keys question is the query sent to the the rag application answer is the response and context are the document chunks available to the model a perfect faithfulness score of one indicates that the model's response could be fully inferred from the context provided context Precision measures how relevant the retriev documents are to the query a context Precision score closer to one means the retrieve to context is highly relevant the only change we need to make to the faithfulness valuation chain is to import and use the content Precision metric instead the context Precision chain similarly takes a dictionary with question context and ground truth Keys representing the input query the retrieved documents and the ground truth document that should have been retrieved printing the results we can see that we achieved a high context Precision indicating the retrieve process is returning highly relevant documents in this chapter we'll discuss how to use a graph database in our rag architecture to replace Vector storage and retrieval the rag architectures we've explored involve embedding a user input and querying a vector store to return relevant documents based on their semantic similarity although powerful this approach does have some limitations firstly document embedding captures semantic meaning but struggles to capture themes and relationships between entities in the document Corpus moreover as the volume of the database grows the retrieval process can become less efficient as the computational load increases with the search space lastly Vector rack systems don't easily accommodate structured or diverse data which are harder to embed we can address all of those challenges with graphs graphs are great at representing and storing diverse and interconnected information in a structured manner entities like people places and sports teams are represented by nodes and relationships between entities are represented by labeled edges notice that edges are directional so relationships can apply from one entity to another but not necessarily the other way around we'll look at this more closely in a later video neo4j is a powerful graph database option designed to store an efficiently query complex relationships throughout this chapter we'll integrate near 4J with Lang chain to build a graph rag architecture let's return to our example and build a near 4J graph using this information our entities are represented as nodes where the color indicates the entity type such as the person the relationships are represented by edges with types like located and interested each node also has a type and unique identifier nodes can contain any number of properties represented as key value pairs so how do we go from the unstructured text Data we've seen to a nice structured graph there's a few ways to do this but we'll be using llms let's load the Wikipedia results from searching large language model using the Wikipedia loader class and split the first few documents into chunks each document has page content and metadata as seen here We Begin by defining the llm to use for the transformation and use it to create an llm graph Transformer note that we use temperature equals z to produce more deterministic graphs for greater reliability the LM creates structured graph documents by parsing and categorizing entities and their relationships which it infers from the documents the transformation is performed using the dot convert to graph documents method on the documents in the output we can see how the model inferred many entities from the text and created nodes with IDs and types to match relationships between the entities were also inferred and mapped using edges going from a source node to a Target node now that we've created our graph documents we'll need to store them for query we'll be using neo4j to store and query our graph documents we won't be going through the neo4j setup in this course but feel free to check out the link shown for information on how to do this new 4J has both cloud-based and local versions to suit your use cases for our purposes we'll