Breast Cancer Detection with Python

Key Takeaways

[Music] what is going on guys welcome back in today's video going to learn how to detect or classify breast cancer in python using machine learning so let's get right into it so let's get started with some basics and the first thing we need to do is we need to install a couple of external libraries that we're going to use to load the data set that we're going to use to pre-process the data to train a model to evaluate a model to visualize results we're going to need a couple of external libraries for that and in order to install those we open up a command line so cmd on windows or the terminal on linux and mac and then we type pip install and we're gonna need numpy we're gonna need pandas we're gonna need matplotlib we're gonna need scikit-learn for the machine learning and for the data set and optionally you can also install seaborn which i'm going to use to plot a correlation heat map i'm saying optionally because we're not gonna uh you don't have to use it because that's not the focus of today's video to plot a correlation heat map i'm just gonna do it on the on the side um the main focus is on getting data pre-processing it and then training a model and doing some predictions or evaluating the model so for those things we're going to need those libraries and this is going to be an additional library that we're going to use for a more advanced visualization so once we have that we're going to start by saying from sqlearn.datasets import and we need the load breast cancer function now at this point i think i need to mention i think it goes without saying but i need to mention that this video is not medical advice so i'm not claiming that you can um use this machine learning model that we're going to build today to do actual medical work this is just programming advice so don't take this as medical advice the basic idea is that we're going to have a data set and i can load this here into the script by just saying data equals load breast cancer and then i can just print data and the basic idea is that we have a data set with certain information about tumors and we need to say if these tumors are malignant or benign basically bad or evil or harmless you could say i think this is kind of what what this means um and to see what all this data is about we can actually go and say data.keys to see what kind of keys we have here and we have the data itself which is the information that we need to do the prediction then we have the target which is the prediction zero or one or malignant or benign um then we have some stuff that i'm not gonna look at and then we have target names and feature names target names basically malignant benign and feature names basically means um the features that we have so we can actually go ahead and say print data and then we're gonna get the feature names and we can also do the same thing for the target names so target names feature names and you can see here that the feature names are mean radius mean texture mean perimeter uh mean area and a bunch of other things and then the target names are malignant benign so we use all these things here all these measures so we can go ahead if we have the necessary equipment we can get such a tumor and we can measure all these things and once we have them we can use them um to make the prediction or we want to be able to use them to make the prediction so essentially you could say from a machine learning perspective the features are the x values and the targets are the y values and because of that we're going to say now that x equals data and then data and data contains basically all the feature data without the column name so those are the column names you could say and data data is essentially just are the numerical values that belong to these columns and y is just going to be data target so now we have this data but usually machine learning what we do is we split into training and testing sets so we don't just want to get the data and then train them all and then we have a model we want to see also if we train them all on some part of the data how well does it perform on unseen data and for that we're going to say from sqlearn.model selection import train test split and then we're going to say now x underscore train x underscore um x underscore test y train and y test is going to be equal to train test split x y and the test size is going to be 0.2 indicating that we use 20 for testing 80 for training we take 80 of the data we train a model based on those 80 and then we try to see how well it performs how good the predictions are if we apply it on 20 um of the initial data which is the testing data set now all we need to do here um to actually build such a model once we have the data is we need to just say from sklearn dot and now we can choose a model that you want it should be a classification model so you can go with logistic regression you can go with a support vector classify you can go with a neural network if you like i'm just going to use a simple k neighbors classifier so i'm going to say from sklearn.neighbors we're going to import the k neighbors classifier you can also use random forest and maybe in fact you're going to get better um better results by using something like a decision tree or random forest classifier and of course you can also tweak hyper parameters which is a bit more advanced so this video here is targeted mainly at machine learning beginners because this is a quite easy task so we're just going to use a plain k neighbors classifier i'm going to say that clf equals a k neighbors classifier and now all we need to do in order to train this classifier on the data it's going to say clf dot fit x train and y train this basically means that we take the data um basically train test split takes the data and then randomly picks 80 out into the training data so into x train and y train and then the rest um the 20 the remaining 20 are um y test and x test and now we build that classifier we train it on the training data and now we have this classifier that is able to make predictions this is already happening all behind the scenes we don't implement this from scratch and if we now want to know okay how well does this model perform on unseen data on x test and y test what we do is we just say print and then clf dot score and we pass x test and y test and what this basically means is that it's going to look at the test data it's going to make a prediction and it's going to say okay is this prediction the same as the y test prediction so it's going to look at the first data row and it's going to say okay based on this i predicted this is malignant or benign then it's going to compare with the y test so it's going to compare with the actual um with the actual truth so is this malignant or benign and then it's going to say okay was this a correct one or not a correct one um not a correct prediction and then in the end we're going to get a score which is the accuracy how much percent of our predictions were true so we can run this now and we can see here 91 now if i run this a couple of times we're going to get different results 92 92 95 94 92 96 94 so you can see we got different results because we do a different split every time we can also fixate uh or we can also fix that uh random state by just saying random state equals and then a number for example 10 and then we're always going to get the same split so we're also going to get the same result 0.921 now this is how you do that this is how you evaluate the model and now all you need to do theoretically to actually predict new information is you take all your medical tools obviously we cannot do this on the computer now we don't have an actual tumor to look at but if you have a tumor and you have the measuring devices as a medical professional you measure all these features so you look at the features like what did we have mean radius and asymmetries and all that you take these measures you put them in a document you write them down and then what you do is you go and say clf.predict and then you feed in here the x new data which is the data that you have so let's go ahead and generate some random data so that you can see how you would do a prediction once you have data now of course if you have the actual data if you do the measurements you're not going to generate random values so you would do something like x underscore new equals and then in the list you would pass all the values so you could go ahead and print data feature names again and when we run this now what's the problem here okay let me just remove that you would do that and you would see here the individual features you would do the calculations you would do the measurements whatever you do and for each one of those you would say okay x new equals and then you have a list here and you would say okay the mean radius is i don't know i'm going to pick now any value 3.4 mean texture 1.2 i don't know you just fill up these values with with the values that you calculated or measured as a professional or you take them from somewhere else if you have the data already and you put them in a list like that or you load them from a file and usually you want to use numpy array so we're going to say np dot array and we're going to turn that list once we have it into an array and of course for that we need to import numpy so import numpy s and p um and once you have 30 values you essentially just feed them into the predict function and you get a result so here what we're going to do we're going to also import here random and we're going to say that when i do random dot sample and we're going to use the range 0 to 50 for example we're gonna take five values from there not five sorry 30 values because 30 we have 30 features um and you can see how many features we have by just saying length of feature name and then you get 30 as a result here so here we generate now 30 um 30 random values so meaningless values essentially but if we have a list of 30 values in practice of course not meaningless we go ahead and we say print clf dot predict and then we pass a list why do we pass a list because usually we predict multiple values at once so we don't just pass this single instance into the predict function we pass a list containing that instance so x new for example uh and in this case it would say yes it's malignant uh or actually one was benign right i think one was benign so it would have have the class one and to figure out which one this is you can just go ahead and say uh data and then what was it target name right target underscore names index whatever we get out of this year and i think we need to actually say 0 because i think we got a list as a result and then you see benign okay so this is how you do a basic prediction now what we can also do is to make things a little bit easier for us we can visualize certain things and this is what i said with optional add-on because that's essentially here we're already done with the main topic of the video we load the data set we train a model we evaluate it and then we make predictions this is all it is but i want to show you a little bit more um interesting stuff that you can do with pandas data frame and with correlations and with the visualization uh using a heat map so what we're going to do now is we're going to take the data and we're going to turn it into a pandas data frame so we're going to actually need to import pandas by saying import pandas spd [Music] and essentially what we want to do is we want to define the columns so we want to have a data frame which is like an excel sheet you can think about it like that and you have the columns which are the features with a column name at the top with the values as rows and then you have the target which is the prediction like or the classification malignant benign and this is what you want to have and then you want to calculate the correlation so which feature influences this final feature the most and why and how negatively or positively a lot or just a little and for that we're going to start by defining the column data and the column data is essentially going to be taking the numpy arrays and concatenating them because now we have x y but we actually want to have one big array with all the columns because the target is also column so we're going to uh np dot concatenate and we're gonna concatenate the data uh data and the data target now i'm not sure if this is going to work just like that i think we're going to have some issues let's see column data yeah we have some issues so what we need to do here is first of all um specify axis equals one but i think that's not going to solve the problem uh got multiple values for argument axis oh i think i'm passing i should pass this as a list right [Applause] that's not the problem there you go okay so now we have a different problem we choose axis one we wanna merge these two but they have different dimensions so what we can do here is we can actually say um colon and none because this basically just has one uh such one such column whereas here we have 30 columns and in order to solve this we can just take this one column and concatenate it into the other one or with the other one into a new numpy array so here you can see now that we have the features that we had before and here in the last row you can see only zeros and ones which are the classifications malignant or benign and now in order to get the column names we're gonna say uh do we need an np concatenate for this one i'm not sure we're just gonna use it np concatenate and we're gonna take the data uh feature names and we don't wanna get the target names because remember we have one column with zeros and ones we don't have two columns malignant and benign we just want to have one column for the actual classification so we're gonna add to this a list with class i think this should work i'm not sure but i think this should work and now we're going to say df equals pd data frame and we're going to add here the column data and the column names are going to be columns equals to column names and then we're gonna print the data frame and see if this worked no we had a problem because um oh we're concatenating i don't need to add there you go now it works so now we have maybe we should use a lowerca lowercase c for consistency here but you can see here we have mean radius mean texture worst fractal dimension and then class 0 1 and those are now all numerical values so what we can do easily now and this is the plotting of the correlation heat map is in pandas in order to get the correlations we can just say dot core like that so i can go ahead now and you can see here the correlations now we have a lot of columns in between here but we can see the correlations between the individual features uh so we have two features compared with each column and row and of course if the feature is compared to itself it has a correlation of one meaning that it's the exact same thing um zero would be completely random negative one would be completely opposite and everything in between is a tendency you could say and this can now be used to visualize a heat map so we go ahead and we import seaborn as sns and we also import matplotlib.pyplot as plt and what we do now is we essentially say sns.heatmap and we take df correlation so we can delete that as the input here we can choose a color map i think i usually go with what was it cool warm was it like that i'm not sure um annette is equal to true to get some annotations and then we want to say announce underscore keyword so kws equals and then uh font size 8 and then we should say plt type layout to get a good layout and then plt show to get the final result i hope i did everything correctly here there you go so now we can increase this i think my camera is going to block this for the most part um so i don't know where my camera is exactly currently only have one monitor so i cannot look it up um but essentially what you see here i hope you can see the whole thing is we have the features on the left we have the features on the bottom at least the numerical features um and we compare those so you can see here in the middle we have the diagonal off once again the feature compared with itself it's always gonna have the same movement um and then you can see also some more negative stuff so the more blue something is or the more dark blue something is the more negatively it is correlated so if some feature rises and another falls at the same time roughly that is a negative and inverse correlation if two features rise and fall roughly the same not exactly because that would be one but if they have roughly the same tendencies they are positively correlated so something like zero nine one here is an almost one correlation or zero nine seven even which basically means there is some difference but all in all it's essentially the same thing or at least kind of the same thing um so for example the worst radius and the mean radius are highly correlated as you can see it makes sense the only thing that is interesting for us now is the class because we don't care about all these features they're going to have some correlations or not um the important thing for us is how do we predict the class now remember that zero means that the tumor is malignant so bad and one means benign which is good for us or for the patient so essentially a negative correlation means that when a certain value rises the class falls so in this case when the mean radius rises this leads to the class falling and in this case this would mean that it goes closer to zero which means closer to malignant which is bad so the higher the radius of the tumor the more likely it is um malignant and we can see here a couple of features now if we have something like zero uh eight and i think that not all features are displayed because we don't have enough size or because the layout is not enough i think if we disable maybe the tight layout we can actually see more of the features is that the case uh not really now there i'm sure there's some settings to to do this a little bit better but essentially again what you see is that some values some features are having a pretty low correlation for example this one here has 0.06 or something so it's basically irrelevant for the prediction whereas some of them are highly negatively correlated you can see that none of those has a positive correlation or a significant positive correlation which means that every feature that we have here is basically a negative feature the more this feature occurs the more likely the tumor is actually malignant and it's better for all these features to be smaller at least numerically um this is just some analysis here that's quite interesting to look at because as a result of that you can for example throw away this feature this feature this feature this feature and focus on the most important features to make the classifications uh if you want to optimize the process uh and yeah this is how you detect or predict or classify breast cancer in python so that's it for today's video hope you enjoyed hope you learned something if so let me know by hitting the like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this channel and hit the notification bell to not miss a single future video for free other than that thank you much for watching see you next video and bye [Music] you