Deep Dive into Text Splitters in Langchain: Character vs Recursive Character Text Splitter

welcome fellow Learners when you want to extract answers from a large Corpus of Text data using a large language model you typically use a retrieval augmented generation or rag approach the first step in this approach is to efficiently split the large chunk of text this is where character Splitters and recursive character Splitters comes to the rescue these tools are essential for breaking down text into manageable pieces especially when dealing with large documents or processing text in a smaller chunks for task like summarization or even training large language models in this video we will explore how these Splitters work by analyzing their source code and examining their output by implementing them in Lang chain all right let's start by looking at the character Splitters first so this is the whole chunk that we are going to pass to our character splitter and recursive character splitter and see how does those Splitters perform so this is basically five paragraph uh text Chunk that we are going to utilize so let's first import this character text Splitter from Lang chain so we can import from Lang chain. text splitter we can import this uh character text splitter once we have this character text splitter we can create object of this character splitter so we can uh basically pass different parameters to it and then then uh we can create this so few of the parameters that we can pass is like chunk size so chunk size is basically the size of the each document that we want to split this whole text Chunk into so let's start by 100 we can also pass like chunk overlab this is basically uh the count of Chunk that we want to overlap in the next Chunk from the previous chunk so let's uh keep it to zero at start and then we will see it in further steps so now we are having this character splitter so the next we can utilize create document function from this uh character text PR to basically uh split this whole chunk so we will use this create document function from it one more thing is like this create document function need a list as an input so we will create this list until we are having a number of uh input with us so as for now we are only having this one text uh chunk so we will be uh creating and passing this as a list once uh we have done that we will get our all uh splitted documents we can print those documents so yeah so so you can see these are the documents have been created this is the first one uh this is the second one and similarly third fourth fifth one one more thing that you would you would see is like there is a warning four times there is a warning like created a chunk size of 439 which is longer than a specified 100 uh so as we have already defined the Chun side should be not more than 100 but still it is creating Chun size of 439 so what's the problem here so if you see for this uh character splitter we need to have uh separators okay so first let's see what is the separator here so if you see uh this is the separator back SL and back which is mostly treated as a new paragraph separator so if you go to the source code for this character text spitter so you can uh go to the source code uh from where you have installed this uh Library this corrector text library in Windows you can find out it uh into the environment let's say I have created in this my SG face uh environment you can find it out under side packages and then the Lang text printers so there are two main file over here base and character so I have basically opened both of these two files so if you see in this Corpi file both functions are there both classes are there character test spitter and then basically recursive character text splitter so let's first see this character text splitter so if you see uh this is the character text splitter and it is basically inheriting this text splitter class so this text splitter class is basically coming from this base py file so this is the text splitter class that it is being inherited so if you see uh in the second line of code we are using this create document function to basically split these uh text Chunk into different uh n number of documents according to the chunk size but if you see there are only two function in this character text bet in it and split text so it must be using uh that create document function from its uh Base Class that is text spitter so you can see this is the text spitter class and there is a function named create document over here so what it does this uh create document function basically takes takes all the text so as we as in our case we are only having one text Chunk uh but uh as we are passing the list to it it requires list as an input it basically takes all those list of text Chunk and then enumerate or iterate over it to basically split split those text Chunk into different number of chunks so so it first uses this self uh do split text to split all those text so if we go to this uh split text function uh this this is basically an abstract method so it must be implemented in in its basically inherited uh class so here in corrector text splitter they have implemented this split text and they have created different uh they have utilized different functions over here so the two functions are split text with Rex and another one is merge split so they are using separator over here so in our case as we have seen the separator is back slash and back sln so that will be utilized and it will just simply split this uh whole text into the that back Lon back Lon so as we have seen in the data it is basically consist of these five paragraph data it will uh split this text into these five different chunks uh on the basis of these five uh these basically four you can say uh new paragraph backl and backl and separators it will split that and then it will apply this merge split function as it is implemented in this uh base uh class this is text splitter there is already a function or you can say a method already been implemented this merge spit method already been implemented if we go to this merge spit uh method we would found out like this uh split method how does this work it basically takes all those splits in our case like it would be all five splits and if if the length of that split or basically one particular you can say text Chunk is basically greater than the chunk size it would directly add it to the our uh good split uh basically list this current dog or you can say the complete dock part so that is the case happening over here so once it has basically uh splited out in these five paragraph and then it will go to that this m split function and it will see like this 439 is greater than 100 it will directly uh provide us as an output so what we can do here so the first thing that comes to our mind we need to change our separator so to change our separator we can uh use a different separator which comes more often than that backlash back SL as like that back slash and back slash new paragraph separator only comes four times so we could choose something like a space or comma or this full stop as our separator so we can do this uh over here directly so like we can do this like cor SP do separator equals to this space and then uh let's see how does this output changes so then again use this character splitter. create document so copy this and just paste it over here and let's see how does this work great now you can see there are different number of documents have been created like you can see the first one is uh a lot smaller than the four documents or five documents that has been created over here and you can see uh these are the documents have been created so uh so let's print out the length of these documents so let's say uh print out this for dock in Docs and uh we can print length of each documents so length of this uh for each of the document we can print uh this page content is already a string so we can get a length of this string directly do page content and you can see like it is uh 100 it is less than 100 every time so once it is more than 100 it will basically uh jumps to this uh next document or basically create next document and if it is close to 100 it will basically add it to the um basically the final document part so this way we can see that we need to create a good separator we need to basically use a good separator which comes more often such that we can create a good number of splits using this character text splitter so now let's see how does this uh character text splitter make changes to our input uh big chunk input text Chunk so what we can do we can basically uh color code each of these document into different color such that we can see the difference like at which character it is basically changing so for that purpose uh we can utilize a library named term color so it will basically convert the string to a different uh color format so we will be importing uh two modules from it colors and colored so this colors model is basically having different different uh number of colors in it there is a dictionary and it is having different colors Let's uh use the different colors from it and convert it into a list so let's uh convert this dictionary into a list and take keys from it so we can do this by colors. Keys uh once we are having this uh colors uh what we can do we can again iterate over uh each of the document in that uh output document output chunk document so this uh we can do and uh then we can basically use the colored module from that term color library and then basically color each of these documents so for each of these document we can p the F the page content from this as it is a string and then basically use this colors list from it uh one more thing that we need to consider is uh the length of this uh document uh would be different from the length of this color and it might be more than the length of the colors so what we can do is we can basically create a cycle so like uh we can use this uh cycle like once the length of this dock increases more than the length of this color it will again go back to black color or basically the zero index so for that part we can do is like we need to use the enumerate to F the index of this each of this to also so do that and then we can utilize this index and then we can basically take the remainder of this index by dividing it with the length of colors so in this way basically it will uh never uh give us the uh out of index out of bound error and it will convert this into the uh particular color of this uh using this Library uh one more thing we can do is like we need to create output test here so output string where we will be seeing our final output so let's do that and then append at each step or for each of the documents and let's see how does this gives us the uh output for final final output string so one more thing like we need to print it use the print such that it basically format it yeah so now you can see it has given us uh this output so so uh like the first uh let's uh change this black color we should not use this black color as it is not visible uh clearly so let's uh uh print it out this color function and uh only use this uh color till gray or yeah not even gray Also let's use it till red only so use this and then we can do something like this we can only fetch this gray color till a red color okay so now we are having this color colors so instead of this colors we will be using this color and then length of this color and yeah so now we are okay so yeah you can see it now how it is basically changing our input uh uh large text Chunk like you can see till this it is in one chunk and after this it is in uh next chunk till here but uh the thing you would see is like uh this particular uh chunk is just splitting it irrespective of irrespective of its context from the sentence like this sentence completes over here but it is still uh chunk it out over here as per the number of Chunk we have provided so that creates a problem when we are when we will be using this splitted documents in the downstream task like any uh rag based approach or something so there must be a something better way to do this so yes like we can use recursive character text spitter to make this uh uh splitting in a better formatted way so let's see how does this recursive character text splitter Works in this case so let's first import this we can import this recursive corre text fter from the same module lch do text splitter we can import this uh recursive corrector text splitter and again we need to create object of it so we will name this just like splitter and uh we will uh create this recursive cor text SP uh we can again pass the same uh argument like uh chunk size uh this chunk size as in 100 Let's uh take same chunk size as chunk overlap uh overlap this one and uh we can keep it as zero as we have done for the previous character text splitter now we are having our splitter let's first see what are the different separators that we are having over here so as you remember the earlier we were using a space as a separator as the back slash and back slash and new paragraph separator was not giving us good result so so in this case let's see what are the different separators it is using so we can do this by splitter. separators now you can see there is a list of separators that this recursive character text spitter uh is using so if we go to its uh source code so we can uh go to this corrector py file as we have seen in this cor text splitter there is another class implement Meed recursive character text spitter and if we go to this recursive character text spitter and in this split function of this uh we would see like it would be using each of the separator and how does uh this work like it will first split the whole uh text Chunk into by using the first uh separator so let's say in this case it is uh this uh back sln back sln this paragraph new paragraph separator and once it has done that uh if you see in this code here over here once it has done that it will first check if the length of that split or the basically splitted each of the split is less than the length of the chunk size or basically the length of the chunk size if it is less than that it would make it as a good split as it has already been uh use this first separator and if it is more than the chunk size what it will do it will basically recursively call again again the next separators and then again split uh this particular chunk to further Downstream chunks such that it can get the better good results from from this chunking so it will do this uh P cursion using these uh different different separators so let's see this how does this gives us the result so again use that create document function so let's use that again again docs we can use this splitter do create documents then we need to pass that uh text as a list also uh we have done that and let's uh again we can use the same code as we have already implemented this code and let's see how does this gives us the colored result to so yeah so let's see now you can see uh one thing you would notice is like it is not basically separated as we have in our original text as you can see original text is uh separated by different uh new paragraph separator so what we need to do is like uh we need to pass you arguments over or arguments over here so one is basically strip white space so by default this parameter is uh true so if we keep it as true it will basically Bally remove all the white spaces at the end of uh the each of the chunk which has been generated through this split so we need not to do that and another one is keep separator so what this keep SE separator would do uh by default it is false so it would not keep the separators like back sln back sln or sln or space at the end of the each of the chunk after the split so we need to make it as true such that it would basically keep those separators in the split also so uh let's run this uh one and let's see how does give this give us result so I think uh this should work yeah this time it worked one thing you would notice is this time it is more uh better output like in in previous one you would see like it is stripping even in between a word also so that would not see in this case uh it is split is splitting with respect to spaces but not even uh not with respect to the word only so what has changed over here like this is basically a recursive character text splitter so it will go through recursion and it will basically split through these one so the thing is like we are using chunk size as 100 over here so once let's improve uh basically increase this chunk size and let's see how does this uh make changes to our output so let's use this and then copy paste this one directly and let's see the results so uh let's use this chunk size to something 500 or 600 something so let's say 600 or something uh then yeah so now you can see like chunk size is 600 and it has basically perfectly uh chunked out this document into uh this complete whole text Chunk into different documents which are basic basically separator of these paragraphs so what we have we can see over here is like this uh way with recursive character text spitter we can get the better results with respect to the character text spitter so whenever you are in a hurry you want to implement a text spitter in your application like if you are implementing a rag application and you want a corrector text spitter uh never start with a corrector text spitter always start with a recursive character text spit and you would get a basic basic results so uh there are other other text Splitters also like uh like you can use document Splitters also where you would be like using uh markdown splitter so let's say there are different other input types like there are lot of image input there are lot of document PDF input there are a lot of inputs like HTML or markdown inputs so you can use those documents l also where you would be basically basically creating these separators based on the type of document so let's say you are having a python code with you so those separators would be kind of the class the keyword would be class or basically the different other one like break or something those kind of keywords would be there so instead of these separators these English text separators you would be using those particular language separators other separator other splits are also like semantic separators there you can utilize embeddings to basically split out the whole uh document so you can use any uh embedding model and then convert these whole uh text Chunk into different embeddings and then basically uh find out the similarity between different chunks so that they can be kept separate or kept with each other as as one chunk so in this way you can utilize these uh these different uh Splitters so um my suggestion is to basically uh use these basically use it in different examples use it with different examples test it out and see how does this performs I hope you got a good understanding about character text sper and recursive character text spitter in this video uh thanks for making till this till the end of this video goodbye until the next time