Kaggle Competition- Implement A DNA Classifier using NLP

Key Takeaways

hello world today we will be working on DNA sequence sequencing with the help of machine learning now machine learning is being used extensively in medical science let it be if you know machine learning and deep learning both are basically being used in medical science suppose if you want to predict whether the person is having a cancer or not so machine learning is being extensively used proper algorithm then trying to reduce down the error also of the predicted outputs today it will basically be understanding how we can do DNA sequencing now usually you know that our DNA's human beings the basic idea in human beings easily consists of 80 ad GC or different kind of you know sequences that are basically present so here I just came across a wonderful data set in Kaggle and from that I've basically taken up this particular project and what I've done is that by using DNA sequencing I basically applied a classification algorithm which will be able to you know classify this particular sequences in human like what kind of what kind of gene class they belong to okay again you may be confused about gene plus I'll just show you different types of gene bugs that are basically present in this particular dataset but make sure you watch this video till the end because you will get to know a lot of things because I also learned a lot of things from this and so I prepared this thing for you so to begin with what I'm doing is that I downloaded the data set that was present in cagin and I'll be uploading all this this to Patel notebook and the data set in the I'll be uploading it in the github and the URL will be provided in the YouTube description box of this so initially to begin with I have some of the new libraries that I will be using like numpy pandas an octave and after that I have the data which is called as human underscore data dot txt now when I read this particular data set I basically have sequences and another feature that I have is basically class okay now based on the sequence we should be able to predict what class it belongs to now this sequence may be the gene sequence or a DNA sequence of a particular human being and they are basically classified into various classes and this particular classes I'll just show you this particular information apart from this particular data set I also have some data set from chimpanzee data and jantoc data which I got it from kaggle itself and this particular data set will also be having the same thing which is called a sequence and class so what I have done is that I have basically applied a classification model after applying the any sequence in human underscore data I would like you to please perform a please implement for chimpanzee data and all underscore data okay so please do it because you'll learn a lot of things in now to begin with after I read this particular data let me just show you what kind of classes I basically have so if if that particular gene sequence belongs to class zero this is basically of a gene family called a g-protein coupled receptors whereas if there is PLAs naval as one on that is basically polish tyrosine in s and I'm not British nor about all this gene family because I'm not an expert II but this particular data set you know I just retrieved it and I just saw what all this fuss is when I just try to show you way or through you and this number basically shows that class label is present on 531 class one student on 534 flash two is present around like 349 reports are present for three six seventy two four four seven one one and four five to forty so this does not look like an imbalance data set it is it is a balanced data set altogether you know some differences will be there but I think we can consider this the value is it aside now what I do is that there is a very very important concept which is called as DNA sequencing now remember I have my independent features something like this I can't just directly give this input features to our model or to our machine learning algorithm and tell them that please determine the class it is very difficult so here I am going to also use some Nash language processing techniques and there is a very important concept which is called as K M ers Emil's came accounting okay and this came accounting you know recently in a research paper you know whenever you are working with DNA sequencing or transcription we basically convert this DNA sequences as languages and in order to convert those into languages we basically use this particular technique which is called as game or counting now in this particular technique what we do is that you can just read this particular details over here the challenge that remains is that none of the above methods results in vector of uniform length don't remember whenever I want to apply some natural language crossing like count of words bag of words basically or tf-idf right here I have a huge list of sequences in each and every two quads right but I don't know like how many vectors I have to consider with okay so what I do is that by using camus counting I make a fixed set of count variable now let me just give you an example suppose I want to use words of length 6 which is also called as examines since this is a k-mer counting process suppose let me consider that from all that sequence I'm taking the word of length 6 okay now suppose this is my sequence over here now when I have this particular sequence if I consider if I consider the word of length 6 okay then k-mers will be acting in some different way you can see this suppose I have all this a a G C a e GC a okay so in the first sequence since I've taken the count as 6 what it will do is that it is it is going to take the first six characters so here it will be 80 g c8 t okay then in the next sequence what it will do is that it it to skip the first one and it will take it from eg C a eg okay so T G a but it see a eg and similarly then it will go with GGC eight eg C then C eight e G C a so similar it will be going till the end now you can see that you will be having different formulation like ATG a CAT AGC AGC GCAT is GC and this on are basically you are trying to form a fixed vector see the vector of uniform length now since I have considered six examples okay six eczema's so what I am doing is that any long my DNA sequence will be I will be converting that into six set of six lengths okay and this will become Z this will be converted into a vectors by using NLP vectors using NLP now you may be thinking why it is done see there is a very good research paper I would also provide you the link of the research paper why they basically used Kaymer underscore king I mean kme are counting so that research paper link is somewhere over here okay here it is now you can just go through this because many scientists how how what is the best way of DNA sequencing that they usually prefer now once they convert this kind of I mean this kind of sequences into a fixed length of sequences then they will try to apply bag of words or tf-idf and convert this into fixed set of vectors okay they'll convert this into a fixed set of vectors now I'll be showing you how that is done but I have suggest that you'd read this whole paragraph which I have put over here okay genomics we refer to this types of manipulation as Kaymer counting and counting the occurrences of each possible k m ER sequence okay and this is the part of because it is very difficult to understand so researchers have come for or come forward with a wonderful approach wherein they are making those sequences into a fixed word of slim and then they are trying to apply they are trying to convert that into vectors now let us go ahead and try to do how we can do it by with the help of python itself so here it is i am defining a function to collect all possible overlapping k-mers of specified length from any sequence string will basically apply the k-mers of the complete sequence very simple now this is my function called as get Camus Here I am just giving my sequence of the DNA and here I have basically given the size and here what I am doing I am converting that whole sequences into this particular size of Kaymer's okay LAN what is hi I will get I will basically get the size of six six six six different different G DNA sequences from the sequence or from the input sequence that I have basically given will just uh you know execute this particular code and see how it will work now you see this I am taking my human underscore data inside my human underscore data what if you see over here just a second okay inside my human underscore data and basically applying this ket k-mers function on my sequence of data now my sequence of data is basically if I go up this is my sequence of data you can see it away again sequence of data now this particular data set will get converted into six set of six six six you know sequences of DNA's with the help of Kaymer's function okay none once I apply this particular function on human data I am also doing it for and I'm dropping that sequence because I don't require it and similarly doing it for the chimpanzee data set and for the dog data set okay so that you understand it and finally you can basically apply that now when you go and see your human underscore data dot head now you see how this particular values are converted all that particular sequence has been converted into a list of or you know list of camus counting values okay my game is counting DNA sequence of the length of six okay and that is how we have converted and why do we do that because researchers have already done Network because they may they need to make into a fixed vector size of all these words then only we can apply count of words or bag of words for this particular data set then the next step is that we will basically be applying SK scalar natural language processing to do the camus counting that we have already done it now we need to convert the list of camus for each gene into string or sentences now we have to combine all the sequences together all the sequences together once we combine then it becomes very easy for us to convert into the bag of for this so for that what I am doing is that I am simply writing a simple code over here all this human text I am converting into a list first and then what I am doing I am just joining the list values with a blank space in between so here you can see all are having blank space okay and this particular data is basically my test data sorry this particular data Y underscore data will be my output data because I'm taking from the : 0.9 use okay human dot I log : 0 because 0 basically has all my classes here you can see that these are all my classes now this is my input features human text will have all my input features ok with respect to records and each record will be combining all these particular words all these particulars sequences like this so if you say human text from 0 here you can basically have and similarly if you say 1 you will have some different sequence ok similarly if you say - you'll be having different sequence ok all this different sequence and you have wireless for data where which specifies all your output ok I imagine you have independent features you have a dependent feature you have your independent feature in the form of strings ok now what all you do in NLP we cannot use directly strings and give it to our model we convert this string into bag of words so in order similarly can do it for chimpanzee and dog the same code I have written it over here now what I am doing is that I am trying to convert by using bag of words by using count vectorizer ok I hope everybody know that I have already created a playlist on NLP that is natural language processing I have explained what is before so what is tf-idf so you can refer that and get to know more about that now I am going to create the bag of words using count vectorizer then this is equivalent to came accountings and here i've tried to or you know creating this count of 8 different different angiogram underscore rain and finally you got the best result by making the count mg gram underscore angel now when I kept this this is my CV I've done fit transform of my human text okay and then I have similarity on it for my excel description and next on this photon okay so here it is I've done this now you can see the shape total you know now the total number of columns are around two lakh thirty two thousand four hundred and forty why the word out of bag of words each and every word will get unloaded into a different column unique word will get converted into a different column so here it is see this this will be converted into one feature one vector this will be converted into another vector this will be connected so all the steps have been done by this conflict arises so we see so many number of columns and for three atoll this X dot shape is for human X and let's put chimp is basically for chimpanzee X and little dog dot shape is basically for the dog dickerson so here it is my four three eight zero total number of Records now what I do I go and check whether my data set is balanced or not so for that I'm just trying and trying to see my human data value counts now you can see over here it is very very precisely all the classes are career you know approximately balanced I know some of the data set are low but other classes are approximately balanced so we can basically use this directly and we don't have to handle the imbalance problem if there is an imbalance data sets on them I have already uploaded two videos either you can do down something and over sampling for this you should prefer doing okay now here it is I'm doing the Train test split and I'm taking 20% as my test size then you can see that my train shape is like this my test shape is like this okay and my dependent feature is Y on this for H which is my human sorry wireless foot data it should be wireless code data just give me a second Wireless for data okay so for wireless core data I have so many columns now you can see this many recorders is my training data set as many recorders is my entire data set then what I do is that I apply a very good multi know multinomial naive bayes classifier so here it is multinomial NB and how did I come to this particular conclusion of alpha is equal to 0.1 I basically applied grid search usually the bridge search will take more time which I cannot just explain you directly over here and the way we have to just wait for the execution to get completed so I have taken the Alpha value as 0.1 okay and this I found it from research or then I am doing I am sizing this multinomial NB that which is mine a bias classifier and I am doing fit on explain and my thing finally I am predicting on my X and this put test and here it is I'm getting my Wireless cook bread now the main thing is that you have a see our confusion matrix accuracy precision recall an f1 score after applying the DNA sequencing see this is very important whether k-mers counting will work or not you can see the example over here okay now here it is I'm trying to see my confusion matrix for that I have imported accuracy score f1 score precision score recalls for everything okay and this is my function for get matrix I have my accuracy score away a precision spall aware because follow a and funds Peruvian have basically returned and printed it over here now you can see that my confusion matrix all the diagonal elements are my right values only there are some errors that are basically happened over here okay my curacy is 98% precision is 98% very polished 98% I have funds for nine tables now you can understand guys having this particular accuracy after doing a DNA sequencing you know able to do it and that is all because of K most counting which actually gives you a proper you know in douching wise knowledge like how that research was actually done and what a kind of creativity they are brought to this particular data set whenever they are doing this and this DNA sequences is being used in lot of you know cost of a lot of applications in finding disease what categories of disease that belongs to now if you can do this DNA sequencing you can basically do a lot of later on I hope you liked this particular videos guys Mitchell and upload all this particular materials in github will be you can download it from there close it from there and yes that was all about this particular session I hope you like this particular video please do subscribe the channel or share with all your friends and your son see all in the next video have a great day ahead god bless you all