Semantic Chunking - 3 Methods for Better RAG

Key Takeaways

today we are going to be taking a look at the different types of semantic chunkers that we can use in order to chunk our data for applications like rag in a more intelligent and effective way for now we're going to be focusing on the text modality which is generally going to be used for rag but we can apply this to video and also audio but for now let's stick with text so I'm going to take you through three different types of semantic chunkers everything we're working through today is going to be in the semantic chunkers library and we're going to go to the chunker's intro notebook here and I'm just going to go ahead and open this in collab Okay cool so first thing I'm going to do is just install the prerequisites so you have semantic chunkers of course and hooking face data sets so we're going to be pulling in uh some data to just test these different methods for chunking and and see what difference it makes especially in terms of latency and and also just quality of what we get out of it okay cool so we can come down to here and let's take a look at our data set so our data set contains a set of AI archive papers we can see one of them here so this is the m paper and you can see there's a few different sections in here already we have the title we have author where the authors are are studying or or performing research from and then we have the abstract so what we're going to do is I mean you can do this or you can just use the full content if you want for this paper it's up to you uh but especially one of these chunkers can be pretty slow and and quite intensive so I I've limited the amount of text that we're using here the other two are pretty fast so it's mainly for that one so we will need a embedding model to perform our semantic chunking semantic chunking at least the versions of it that we show here uses or or relies on embedding models and finding the semantic similarity between embeddings in some way or another so in this example we're going to be using open AI is embedding three small model so you will need an open AI API key or if you prefer not to you can use a open source model as well so if you do want to use a open source model instead so you don't need to get an API key you can just do this here but I'm going to be sticking with open AI okay so initialized my encoder and now I'm going to come down to the statistical chunking method and this is probably the chunky that I would recommend people to use just out of the box the reason for that is that it will handle a lot of the sort of figuring out of different parameters for you it is pretty cost effective and it is it's pretty fast as well so this is generally the one I recommend but we'll have a look at the others as well so we'll start with the statistical chunker and the way that this is going to work is that it's going to identify a good similarity threshold value for you based on the varying similarity throughout a document so the similarity that it will use for different documents and different parts of documents may actually change but it's all calculated for you so it it tends to work very well actually so if we have a look here we have a few of our chunks so we can see that it ran very quickly we have the first one here includes our title the authors and the abstract okay so it's kind of like the introduction to the paper then after that we have this I assume it's probably one of the first paragraphs in the paper can see that then we go on to this the second Point here or the second title here and so on and so on but generally speaking these trunks look relatively good of course you will probably need to look through them in a little more more detail just looking at the start of these it looks pretty reasonable so that is statistical chunking it's pretty easy the next one is consecutive chunking and consecutive chunking is probably the second one I would recommend it is again it's cost effective relatively quick uh but it does require a little more tweaking or inputs from outside so that primarily is due to the score threshold so the score threshold most of the encoders here require different score thresholds so for example text embedding order 002 similar for that model is anything sort of within the range of 0.73 to 0.8 that's usually the sort similarity threshold that you would need to use with the newer text embedding models and then text embedding three small you need to use something much smaller which is why I've gone with 0.3 down here so there's a little bit more of you need to actually input something here and to be honest it depends in some cases the performance can be better I I think but a lot of the time it's actually harder to to get like very good performance with this one so for example here I can see that it's probably splitting too frequently from what I can see so I may even want to modify my threshold okay and so I've decreased it to 0.2 and it's seems a little more reasonable cool may even want to go a little bit lower but that looks a bit better so that is the consective trunker again using a completely different process here this is essentially creating your embeddings one after the other so so it first splits your text into sentences actually the same for the for well for all of the trunet here they split your text into sentences and then they start merging your textes into larger chunks and they're looking especially this one is looking at where there is all of a sudden a drop in similarity between those sentences and that is how it defines okay this is a logical point to split our chunk so that's what we're doing there and then the final one I want to show you is the cumulative trunka now the cumulative trunka what this one will do is takes our sentences and okay we start with sentence one and then we add sentence two when creating embedding and then we create add sentence three create another embedding and then we compare those two embeddings so we're comparing the embedding of sentences one and two and 1 2 and three and then seeing if there is a big change in the similarity if not then we continue and then we compare uh basically the group of three sentences followed by the group of four sentences we look at those two see if there's a sudden similarity change if there is then that is where we make our split so what you're doing here is you're cumulatively adding text creating your embeddings and then continuing cumulatively adding more text continuing and the result of that is that this takes a lot longer to run it's also a lot more expensive you're creating far more embeddings but you'll find compared to at least the consecutive trunker it is a little more noise resistant so it requires a bit more of a change over time time to actually trigger a split for this chunker so the results tend to be a bit better but I would say either on par or maybe even a little worse than the statistical chunker in a lot of cases but it's worth trying just to see what gets the best performance for you for your particular use case okay so we can see that yes this trunker definitely took a bit longer than the others and let's have a look at what trunks we got so we come up again and I probably should have changed the threshold here but let's start with these so yeah we see generally I think probably worse performance than we got with the statistical chunker but especially if I modify the threshold here and tweak that a little bit generally you can get better performance than the consecutive trunker so yeah those are the three trunking methods that we currently have in the semantic chunkers Library I will also add that there are differences between these chunkers in which modalities they can handle so the statistical junker for now is only able to handle a Tex modality so great for rag not so great if you're wanting to pass video for example whereas the consecutive chunker is really good at passing video and we do we we have an example on that and I will walk through the in the near future but it's something to consider so consective trunker can basically handle any modalities things like the statistical trunker I cannnot and then the cumulative trunker is is more text focused as well so for now that is it on semantic chunkers I hope this has been useful and and interesting so thank you very much for watching and I will see you again in the next one bye