Semantic Chunking for RAG

Key Takeaways

today we're going to talk about semantic chunking and a few other methods that we can include within our chunking strategy to improve our rag performance but first let's begin with semantic chunking and just understand what I mean when I say semantic chunking so when we chunk our documents semantically what we are actually doing is taking how we embed our documents for a rag pipeline how we store them and how we retrieve them and applying that same mechanism to our chunking strategy so rather than saying I going to chunk with you know 400 tokens or I'm going to chunk based on whether there's some new lines so on and so on instead we are building our chunks to optimize our embedding conciseness now the reason that we do this is if we imagine we have some document over here there's a load of text we you know we could just naively trun this in you know using the token count or using different delimiters or we could consider the fact that we're trying to embed each one of these trunks into a single Vector embedding okay so these are our you know ideal end States we have these single Vector embeddings these are single Vector embeddings but most drunking strategies out there are very prone to containing multiple meanings within our chunks and the optimal way of reducing the likelihood of that is to go for smaller and smaller trunk sizes which we should still do we should still go with smaller trunk sizes but using semantic chunking we can find the optimum chunk size to allow us to take some meaning here so let's say here to here and what we do is find that chunk size that produces essentially a single meaning okay so we have some meaning here and it is concise and as I mentioned the reason we do this is if we for example take a slightly larger chunk and we bring that over here and we have multiple meanings right let's say there's three potential meanings of what this text could be about right if you consider like a for example a newspaper article you take different trunks of that newspaper article and each trunk is you know has a different meaning it's talking about something slightly different even if the topic is generally the same and what we typically do with our naive chunking strategy is we take all of those meanings and we try and compress them into a single Vector embedding so it means that okay we're kind of capturing the meaning of all of our you potential meanings here but we're not capturing the exact meaning of any of them we're just capturing kind of like a diluted version of what all of these things mean together so by semantically chunking we can actually calculate the conciseness of our chunks identify the optimum length for keeping a chunk concise and then using that we create our single embedding and that is semantic chunking so let's go ahead and see how we actually apply this in Python so in this notebook we're going to go through the full pipeline so we are going to get a data set we're going to trunk it we're going to prepare it I mentioned there's some other things we're going to do to our trunks as well we're going to embed all of that and then we'll see how we can begin retrieving some of that information so we're going to go through this notebook there will be a link to this in the description of the video and also in the comments below low and the first thing we're going to do is just install the other libraries that we need in order to run a notebook so we have semantic router which includes our truning mechanism we have pine cone which is where we're going to be storing and retrieving our embeddings from and we have hooking face data sets which is where we're going to be pulling this starting data set from so this data set if you watch these videos often you will have seen it before it's a dump of AI archived papers so you know things like the Llama papers a embeddings and you know many other things as well so we're going to come down to here and we're going to initialize our encoder so the embedding model and we're going to be using open AI for this one and we're going to use the text embedding 3 small model we're going to be using this Bo to create our embeddings for our trunks once we have created them and putting them into Pine k and also you know for that chunking mechanism and ideally that's what you should be doing as well you should be aligning both of those models because you are optimizing your trunks for a specific embedding model so it would not make sense to use a different embedding model to the embedding model you're using to create those final chunks okay and we're going to be using this rolling window splitter so there's a few parameters we can adjust here to see a little bit more of what is going on so specifically the plot splits and enabled statistics they're going to show us a lot more information about what is actually happening when we're producing our chunks now the other parameters that are probably important to not here is the minimum uh split tokens so I'm basically saying I don't want any lower than 100 tokens in my splits and then Max I'm saying I don't want anything higher than 500 tokens so we're kind of setting the bounds of where we want this chunking mechanism to function with them so I'm going to initialize that and then we'll come down to here and we'll perform our first splitting on the first document in our archive data set so if we come up here we'll see we have these charts that shows you know what's been happening so as our rolling window of sentences you know goes through our our paper here or our document what it's doing is calculating the differing similarity between chunks and then it's identifying the optimum threshold for the specific model we're using we're using the uh the small text embedding 3 model from open AI so the threshold is pretty small you can see up here on the top right 0.2 and those similarities between our Windows of what you can see with the uh the blue line as we're going through then you can see that a split has been made where we see the red doted line and one thing that's kind of interesting that you can see here is once it gets to the end here there are you know many more chunks and I I haven't checked but I'm pretty sure that this area here is actually the references for the paper so you can see that it's basically splitting uh many times between references because they don't really have that much similarity between them so yes we can see that uh we can come down to here as well and can see the uh the trunk sizes that have been produced so yeah all you know between our 100 to 500 range except with the exception of the last one here which is It's the final one so there was I assume not enough tokens to fit into that chunk but we can see the the structure here and then we also have the splitting statistics at the bottom here if you you want to check through that and when looking at this the thing that you should pay most attention to is your splits by Max trunk size so you you you want more splits by threshold the the more of these you have compared to these the better your basically semantic trunks are going to be the way that I could reduce this like the easiest way is actually just to increase my Max chunk size but the other thing you can do is actually set a threshold so if you if you want to so we can take a look at the rolling window split a class here and we can see that we have this Dynamic threshold value and we can set that equal to false so we go back into our code we set Dynamic threshold fults then we can also set the default threshold value this is looking at the similarity threshold for what defines a split and you can modify that by going to the encoder score threshold and modify find this and we can just increase the threshold number so before we had roughly 0.22 so I'm going to go to 0.25 see what we get okay and you can see that our splits by threshold has increased and the splits by Max chunk size has decreased and we can keep going with that if we want to um I think with the new embed 3 models 0.3 is a you know usually a pretty good value okay so yeah this is starting to look a bit better maybe too many trunks I would say probably too many trunks there actually um but anyway you can do that I'm actually going to you know take that back to what it was before I'm going to set dyamic threshold to true and the other thing you can do is increase or decrease your window size so if I go with five which I think is the default value and try again you can also see the effect that this has on those statistics and you can see here that we we get a better ratio here but what you tend to find when you increase the the window size here is that the like the similarity is averaged over time and I don't necessarily want to always do that I think it depends on like how high level you're trying to be with your embeddings uh in this case I want to be pretty concise so I'm going to actually just reduce the window size again go back to what we had at the start so I'm going to remove that I'm going to reinitialize our encoder rerun this this and this okay if we come down here we can see we have these chunks they look uh pretty good these and these as well which seem maybe quite small even but we can see that statistics look decent uh but really what we want to be doing is looking at the the chunks as well uh so you can see here the first is is it like the authors and let's just see yeah it's just the authors and then it cuts at the abstract so the next chunk here is actually our abstract uh we go [Music] through okay if we come to the end here we see that's basically the full abstract like just the abstract then we go to the next one here and we have it's going into the introduction okay so it's it has the like the paper details and the authors the title and the authors it has the abstract and then the introduction and it's it's you know broken those apart so that looks I think looks pretty good so now we can go on to uh just having a look at what those objects are we can see them here so we have the document split object here uh this just includes you know all the information that we have there what score triggered it like the the split and token count and and so on then if we come down to here so for each one of those document split objects we access the text Itself by going into the content attribute and then uh well yeah that's how that's how splits or trunks now I mentioned before that we're not just going to focus on spanic trunking majority of what we're talking about but not just that I also want to show you some other little things that can do in order to improve the quality of our our trunks both in our embeddings and also for the LM so the first one of those is this so what I'm doing here is I'm actually taking the title of the paper and I'm also taking the chunk which is going to go into the content here and then I'm just merging those together and then we're actually going to use that to create our embeddings we won't feed this chunk to our llm we just feed it into our embedding model because it gives more context to the embedding model as to what we're actually talking about when we include things like the title of a paper or a chapter of a book you know these sort of kind of like hierarchical components so we can create a few of those and we can see what we have so we're looking at the paper on unveiling emotions in generative AI so that can be a really useful bit of context uh when we're looking at you know sometimes paragraphs or chunks that don't mention the overall aim of or or the overall context of where it is coming from so we do that uh we're going to use this function later when we're creating our embeddings okay and then the other thing that I want to do I mention it here is for the llm I actually want to pull in some context from the surrounding chunks so the way that I'm going to do that is I'm just going to number all of my chunks add that ID into my metadata and then just add a reference uh to the the chunks so I'm doing that with this pre- chunk and post chunk field within the metadata here so if I I run this I'll just show you what I'm doing I don't need to type I just want to show you let me show you one of them so metadata and we go zero okay so we have the title we have the content which is our our semantic Chunk we have the pre- chunk and then here you can see our our post chunk there's no pre- trunk for this one because it is literally the first chunk if we go to number one we will see that this content here will become the pre- chunk value okay so you can see that there we have the post chunk so what is coming after and the current chunk now we have that that's quite a lot of text and it's not so efficient when you are soaring everything in like pine cone so what you can do is you you can you can do this you can just s the trunks themselves it's just a lot of extra information or you can just add the IDS right so so this is a it can be a bit easier I think especially when you're looking at storing a lot of data this is probably what you want to do when it's less data it's it's you can probably just put in the text itself it's not really a big deal so yes uh let me show you what we're doing here so the ID for each chunk that we're creating here is actually archive ID plus the chunk number and of course it it's pretty easy to set up the pre-run ID and post trunk ID because we just need to either go IUS one or I + one so we do that and then we build our metadata and then let's have a look at you know a few of these so we're just looking at the first three records from that from the previous document so we have our you know the title of the document and the authors the pre- trunk ID is nothing of course post trunk is this whereas the current trunk is actually zero okay now we look at the next one current trunk one uh post trunk 2 pre- chunk zero then again current two pre one post It's the final one within this set of three so it didn't actually give us a a post trun ID here okay so you know that's the sort of structure that we're setting up for this data set let's go on to actually indexing all of this so the first thing we need to do is set up our Vector index you go to app. Pine con. to get that then you want to come down to here and we're going to set up our our serverless spec and actually I'm using a paid region here so you can actually switch this to us East one and this will be uh it's the free plan of ser L I have mine in Usos SW too so I'm going to leave that but yes you probably want to use this okay so before we create our index we actually need to find the dimensionality of our encode a model so to do that we can just encode like creating embedding uh and then we can see that dimensionality is 1536 otherwise you can just look online and you will be able to find the embedding dimension of your models then we're going to create an so I'm going to call this one better R chunking I'm going to be using the dot product metric here so that's our similarity metric we can use others we can use cosine for example here uh we're actually using embed three from open Ai and then we pass in our serverless spec as well uh we run that and I actually already have my index actually oops I okay I just messed that so better rag chunking I want to initialize that and and I already have my index initialized so you can see that I have this Vector count here actually this is pretty high uh for the full dat set so I'll show you later you can just limit how many vectors you're you're putting into your index the splitting or chunking that we do here it can be expensive because we're creating many many embeddings uh to produce our semantic chunks which down the line results in better performance or can result in better performance but I mean for this example you I don't know if you you want to spend that so I would recommend if you would like to make this quicker you can actually just download the the pre- chunked data set from here so this again it's from huging face so if I come here you can you can see that you can also see all the chunks and everything if you want to check that out as well actually looks pretty interesting so yes uh you can do that if you want to limit the number of records that you're going to be processing and storing in Pine Cone you can do that so if I modify this you can do for example just take the first 10,000 records like that and that will just limit how much you're pulling through that otherwise if you're using this data set what you can also do two pandas and we can go like I lock I think like this and go with the first 10,000 records like this instead uh so let me try and run that yes so that will work uh so okay looking at the rest of what we have here we set the splitter statistics and splits to faults because we don't want to be visualizing them with every like embedding run that we're doing here it's just we don't need to again like this is more for investigation when you're actually creating your embeddings you you don't need to be looking at these there's no there's no need to then you come down here we are creating our chunks we then build our metadata using that function that I showed you before and then the other thing that we're doing here is buring our chunks for the embedding model so where we're adding the title as a kind of like a prefix and then the actual chunk itself and embedding all of that together which we do here and then after that we just add everything to Pine con which I have already done so I'm going to actually stop this now and I can just continue now that we have our trunks stored in Pine Cone we can go ahead and actually begin querying against them so to do that what we need to do is we'll set up like a little function for for querying so I'm going to to say Define query the input to this will be your your text right your your input query and the first thing we need to do is create our embedding so we're going to create our we you know I call it the query vector or query embedding so I'm going to do encode I'm going to put my text into a list here extract that out and then I'm going to pass this over to our pine cone index so I'm going to do index or we should get the context out of that actually so context equals index. query we have xq which is our our Vector can actually Define that here as well so Vector top K make that a little easier to read top K which is how many context we'd like to return how many chunks I'm going to say five oh and include metadata so include metadata I want to set that to true and that will allow us to return like the the other records that we had in there so I'm going to just for now return the context and we'll see what we have in there and then we'll just modify it accordingly so run that make sure it actually works as well so query text I'm going to say what are ehi embeddings okay so it worked that's good it's not in a very good format for what we need so I'm going to take a look okay we have come up to the top here we have matches so we want to go into our matches we then want to iterate through each one of those matches actually let me modify this we want to iterate through each one of those so for match in matches I want to get call M I want to get the metadata metadata equals m metadata and actually we can just go straight into the content so I can get content metadata content the other things that we might want are the the post chunk ID and pre- chunk ID if we'd like to pull those through and also the title as well so I can get the title pre and also post okay we should be able to run that yes then what we want to do is format this data or you know do something with it so one thing that I would like to do is using the preos IDS that we have here I'd like to fetch them from our index so I'm going to do index fetch you see that we get these IDs so I'm going to get those so we have IDs equals and then that's the pre and post pre post let's see what we get out of that just so want to go to the top okay so we're at top here we have our vectors that we turn I'm going to get those so vectors and then we need to iterate through each one of these or we can just get the preum poost content I think so maybe that's easier so I'm going to say these are our other chunks and then going to get the precontent pre- chunk let's call it pre- chunk equals the other chunks vectors the pre ID okay which is going to give us this then we want the metadata then we want the content okay and then we'll do the same for post okay so that should give us everything that we need I think if I do it right I did something wrong ah okay so we have the ID followed by metadata data followed by content don't know where I got vors from here so ID there we go so that should work cool now what we're going to do is you know P all these together we're going to include it in this query here so we have these are our matches from there we get these Okay so we've got a Content title pre and post IDs then we're going to get other chunks okay and then we're going to put all this together in a way that's going to make sense for our llm so that is going to depend on you know the the format that we exactly need so what I'm going to do is something pretty generic to be honest so I'm going to go with uh we have our chunks here I'm going to say I'm I want some of my pre- chunk and some of my post chunk around my my current chunk and I also want the title at the top Maybe I don't know whether that's super necessary but it's fine so we'll do uh chunk equals and we'll go title first or we can even just do this here we put the title then I'm going to put in a little bit of space I'm going to put in the pre- trunk but I just want the end of the pre- trunk not everything so pre- chunk and I'm just going to go with string values here you probably don't necessarily want to do that but it's it's fine so I'm going to go I don't know the last 400 characters then new line I'm going to go to the chunk you don't need to do new line necessarily but I'm going to here just for the sake readability then you want to do your post chunk and we just want to take the start of that I'll do the same so we'll go the first 400 characters okay and that's everything I I really want that and then I'm going to return the chunks and of course we're returning a fair bit here so you do want to be careful depending on the llm that you're using that you're not going over the context window that it's going to be returning and even here I think maybe it would make more sense to do a smaller topk value like three is probably fine another thing you can do is use reranking of course and that will you can rerank uh with more records and then rerank to we get a more ideal order of those results and then just take like the top three that's usually the the sort of numbers that I would I would go with um so the chunk is sorry it should be content here and also we should be doing chunks. append chunk okay and all of this should come back with something so we can try the query I had before it's like what are H eddings I don't know if this is in here or not so eh I I may not have this in the data set so let me try something that I know is going to be in there um so we can go with what are large language models so should be an easy one okay let's see what we get here so we have section three reuse hist work to evaluate large Lage models are computational models have the capability to understand and generate human language LMS have the where are we new teex Bas on a given input engram models Okay cool so it's explaining what llms are there yeah so that's just a like a quick introduction to this idea of semantic chunking rather than you know the the typical chunking methods that we might see elsewhere and also this idea of just adding know some extra context to your chunks like where we added the title to our chunks and how we actually go through a full pipeline uh where we're creating our or taking our data set chunking it um adding extra context where needed and then actually creating our embeddings and then finally at the end there searching through it so yeah that is it for this walkthrough I hope this has been uh useful and interesting but for now I'll leave it there so thank you very much for watching and I will see you again in the next one bye