Machine Learning with JavaScript | Detecting Skin Cancer with TensorFlow.js | Community Webinar

Key Takeaways

so thanks everyone for joining my name is nathan paccini i am one of the marketing managers here at data science dojo and i am joined today by um carol and i am gonna mispronounce your name even though you just told me how to pronounce it it is carol christoski and he is the cto and founder of codet he's going to be taking us through um you know some javascript for cancer prevention through early detection and you just heard us talking a little bit about about the device that that um that he has with him and i know he's going to be incorporating that in his presentation as well so carol why don't you go ahead and introduce yourself maybe a little bit more than what i just did and go ahead and get started sure thank you for introduction nathan let me just share my screen and uh show you some slides um all right i will just not have slides today um so um i will talk about javascript and actually how to use javascript uh to get machine learning to uh to beat cancer uh and obviously as you see javascript is not uh that's you no they're not not the best language to actually build machine learning models so about myself uh it's uh well i do machine learning and medical diagnosis in skin cancer specifically uh for more or less 15 years now um so i started in 2007 2008 when i actually started ibm uh working at the game i mean i did it as a pg as a part of my studies at pg studies but it wasn't at the same time when i actually started my job at working at ibm and you can find some papers on the topic um that are actually available on google's car for example um that's something that you can easily find you can easily also find some some of my lectures some of my talks about skin cancer how to how to how to use machine learning to figure it out i'm not a medical doctor to be clear i am i'm a computer scientist i'm a specializing in artificial intelligence machine learning uh and one of the talk that actually i gave uh last year is actually published as a tedx talk uh actually it is in polish because i'm actually now streaming from krakow from poland um and so uh it isn't paul's shadow but i hope maybe maybe some the next tedx talk will be will be in english it and actually it is about skin cancer so just to just to warm up uh well skin cancer why this topic um first of all it is a big issue globally i mean skin cancer i mean there are different type of skin cancers melanoma is one of it uh it's the most dangerous one because um if it's not uh found in the early stage it's it's it's it's very dangerous for for our health because it can actually end with death um because it's a very very the cancer is very uh progressive it's very um well malignant and but um yeah and especially because well just to give you a medical introduction uh to the topic we have six types of photo six photo types of skin so for example if you if you are blonde uh you have a white skin uh and yeah if you in blue eyes and if you go outside uh and you become red instead of brown i mean your skin then you're on the first photo type the one that actually is the most uh well where actually the probability of getting a skin cancer of melanoma specifically uh is quite high uh yeah just to mention one one more thing if you want to ask any question you can use the q a part tab and add and zoom also if you can use the chat if the answer answers short i will just answer immediately uh if it's a bit longer i will just keep it to the end of the after the presentation so we have six photo types uh one uh is the white skin blonde hairs blue eyes and uh you get red your skin gets red when you actually are on the beach and not brown so the darker the skin is the probability of getting a skin of melanoma is lower so there are different kinds of skin cancers melanoma is one of it that actually is uh is the most popular of malignant skin cancer because it's the most deadliest one but there are also some other malignant skin cancers for example that are white i mean maybe not white transparent let's say so they are even they are even uh more dangerous because it's very even harder to find such a such a such a mole that is transparent or color of the skin uh more or less uh so that's why actually most of the dermatologists also focus on melanoma defining this one because it's just uh very specific um there are some specific patterns that to tell us okay that's that's a cancer but that's just a short introduction about the cancer so why actually use machine learning why use machine learning together with javascript right so how how to combine that i would like to show you uh where we can use javascript it's not that i am a javascript engineer i'm i mean i i well i used to work with javascript actually the web pages were written with you know were created with tables um so it was many many years ago now javascript will definitely have typescript you have es 5 es6 and of the new standards standards of javascript uh so it's a more convenient way now to to really develop with javascript but um actually my what i'm the programming language that i use on database is still python i mean still it's all the time python uh because it's just more convenient to build the models so why javascript i will show you that in just in just a few minutes and that will also give you a better better understanding of the skin cancer no not about the photo types only but um how to uh well what actually we are looking for so let's say we have this device that is used this is the monoscope that i showed you um uh the beginning that's at the moment i mean it's an extension to uh to uh iphone uh there's an extension like this lens here that's uh um this is actually very important because this is actually allow us to take a picture or take take a look on the mo and this is a device that is used by the dermatologist there are different types of of thermo thermoscopes this is one of the smallest one uh they're a big one like like whole computer with with a kind of a something like looks like a gun um quite big one um that the dermatologists also use that's that's well let's say our pocket uh version of the of the mother's cup uh and then i will also show you in the meantime i would like to show you some demos some code samples in python in javascript um to show you the differences and then where to go next with the project with actually where i'm going next so with the with the project that i have built um all right so as i mentioned before i'm doing uh i'm doing i focus i'm focused on the research on skin cancer for the last 15 years um this is a quite big problem globally especially in the u.s uk australia and the country scandinavia all the countries actually the people are there are many people with blonde hairs and blue eyes because this is this is to actually where the risk is the highest myself even i i don't look like a like a blunt guy but i still i have some some cases in my family regarding melanoma uh hopefully i mean in my case in the case of my family everything went well because it was uh the mall was actually cut on early stage and this is how actually it works right if you find it on an early stage you're healthy there is no risk of actually you know getting other type of cancers in your body because this is how actually melanoma works if you if it go if it uh raises the the vessels it just spreads into your body and this is very dangerous because you will get cancer everywhere uh but if you cut it on an early stage it doesn't spread so it's it's it's uh you are fine so uh what i also did uh i we have signed a partnership with different kind of dermatocopy thermoscope and actual dermatologists rheumatologist companies uh to build some much united models uh what is more uh now when it comes to analysis of skin cancer uh when i started the research 15 years ago uh i had a data set of around 50 57 images of malignant cases and not uh that's not a huge data set to really do any kind of research right but but it was 15 years ago in the meantime i i have i have a bill i have built a data set my own data set of about 5000 images now you can you have if you don't if you want to do it for non-commercial use you can download the isic archive isic archives there are some also some challenges um that are related to the this this kind of this this data set that they that they give uh you can easily donate everyone here can download it uh it's around 25 more than 25 thousand of images now of different kind of skin illnesses not only melanoma and that's very interesting because this is a data set that you can really start working on i mean use it for for some from some real research with 57 images that there's not too much you can really do so because of that it's so it's worth now to do to do any kind of research in this in this topic and if you find if you look on the papers there are now plenty of papers doing research on melanoma with higher and higher accuracy sensitivity specificity and even the the models that i have built uh for my phd thesis i actually i achieve with the models higher accuracy higher uh quality matrix than the typical dermatologist um so that's a different kind of research uh when it comes to the way how we how we do that uh how did the mythologist find out if it's a cancer or not but that's something that i will show you in just a few minutes when i when i will show us some pictures uh how it looks like if you want to use javascript for machine learning i have built a docker image for you it's it's quite old but it's still it's still um valid there is a javascript kernel uh for uh jupyter notebook uh and all of the javascript libraries that i also have here as a jupyter notebooks on github that you can use easily so feel free to download javascript libraries uh for machine learning so how i started this research uh i started with using different kind of because as you can see this device it's uh well this is an extension kit to an iphone so it's easier to build an application uh that you use a model on an iphone using javascript instead of python i mean technically it might be doable to write a python application for iphone but actually it's much much easier to actually use javascript because you can use for example now actually react native that's the most popular solution for that in the past i use also phonegap i use cordoba and x react and some other some other frameworks and libraries to actually make this application uh available for this mobile uh for more for my mobile phone um that's actually an iphone 6s so it's quite old but still working well for um for skin cancer recognition and obviously uh for the models uh i i use i i'm using uh tensorflow.js um well why tensorflow.js are not a framework that's a very important question to set what kind of libraries you should use or what actually you should not use when you when you work with javascript so why javascript one of it actually one of the reasons is clear that's because you can easily build our application to use the model and because tensorflow is written in such a way actually you can use the models with python but also on javascript also in java and some other languages uh because the code is actually written in c plus plus and you have you know also you know you have also some um some libraries that actually use that but actually libraries written in different languages like python is the most popular but you can you also have uh it's written in javascript and what is important to mention here that actually it is robust i mean tensorflow.js is robust compared to um to other labor libraries i'm talking about javascript now only right because in pi we have plenty plenty of libraries that are well designed well implemented and uh robust and you can easily use it in production um so anyway uh what i also have what what is also the advantage of using javascript uh for my for machine learning is that actually you can use it easily for for web applications right so if you want for example want to move from your mobile app to your web app uh using for example your camera i mean it doesn't make sense for skin cancer right but because because of you know you don't have to just kind of lenses on on your on your camera camera on your laptop in your laptop but if it doesn't actually it's if it's a model that doesn't use this kind of a lenses this kind of a topic you can easily move it to a web app and you used also there so how would you look in production this is a very very uh general let's say a very high level uh way how it might be used uh really there are two ways one of it is actually where actually you use this model as a part of the app so you have an app that's actually a javascript and you have also a model that's loaded and you can easily use it or what you can do you can also use a tensorflow service um well i don't want to go too deep into the details i know that the mlaps as a topic is quite popular now and it's become more and more popular but the tensorflow servers serving actually the the model is is easier to do it with with tensorflow in my opinion uh so you can actually have more more apps that actually will send the image to the servers and actually do the evolution on the server it it depends on the internet connection so that's the disadvantage of this so kind of a solution in my case i have both so uh what i do i do the evaluation on the the first evaluation on the app and if the doctor wants to compare two different ones like no oh i see it's suspicious right so if i want to compare i can also compare it using the web web service uh the ts the tensorflow service and actually try to compare with other ones and get a more more precise explanation of the of the prediction okay so let me show you a demo um here we go when flutter um flatters you know flutter is not javascript um so i think that's not the topic for the uh the talk today uh i hope you can see my screen now again uh so i did this workshop a couple of three years ago when i was when i visited hockey if hakeem is the city in the east part of ukraine and i did a comparison of the libraries and available libraries in javascript and titan you cannot easily find this notebook on one of my github repositories what is important here obviously in in the way how to how machine learning or training works that's here explained but what's important is just to go through the libraries you actually can use in javascript and actually since then i mean since i gave this talk a couple of years ago not so many uh there are there were not so many changes really when it comes to the javascript word and machine learning so when it comes to data manipulation uh if you talk uh if you ask our data scientist or machine learning engineer uh they will tell you okay there is only one uh library for that and it's called pandas right there is uh there is a library also in javascript that is called called pandas pandas gs really here as you can see here you have series you have data frames so you can easily work with the with the with a huge number of of data also in javascript but it's not well i mean for the for some basic operations i think it's uh it's worth considering uh this as a library to do that but it still lacks of many many uh many functions uh that actually the original pan as a python pandas does have so that's that's something that uh you you should consider that actually that's not idea of the best solution but if you are if you are forced to use javascript for um for actually doing some data basic data analysis then probably you will choose pandas js or maybe data frame js there is recline data forge as well there are some links here you can test it out but you can see you can easily work for series like here i mean the output here if you compare it to python has different types it's it's uh i in my opinion did that done in a let's say uh more user-friendly way compared to javascript uh oh you can work with a data frame here's an example how to actually work with data frames you can convert it easy to json that's actually easier i think compared to python to the python version or to string you can easily work you know change some do some filtering on data frames so some basic opera functions are there still it's still limited um so probably if you're not forced to actually use javascript you probably used to use the original pandas still still that the pandas in javascript is not yet such robbers i mean in general when you compare the libraries in javascript and python and see the numbers so in javascript you have more libraries for machine learning than in python but the thing is that if you if you if you if you take a look on the robustness of how good the libraries are then the python are obviously the even less they are just better compared to javascript ones because there are plenty of libraries in js that just doesn't work or are not yet to make are not maintenance now or uh you know someone did something i mean develop some a small uh small method and then just you know is doing nothing with it and if you for example compare it to cycle learn that's a huge hugely different because everything is i mean when it comes to not shallow metals everything is cycling then when it comes to visualization uh i think javascript has disadvantages that just done a bit better compared to python probably some of you might disagree with me because you have model lib you have cbr and some other libraries but in my opinion it was in my work it was a bit easier to use the javascript i mean the javascript libraries for chart drawing charts uh i and are more convenient to use using kind of d3 for example uh re-adverse recharge and there are plenty plenty of other libraries that you can use for drawing in in js uh when it comes to the machine learning again and the the the shallow methods right and so you have um i think actually a port or let's say a library that should work like cycle learn that i mentioned before but there are two that i mentioned here that js kit learn and cycle learn in js uh but both are not uh not yet not maintenance anymore i mean one of it's actually the last time that stopped it was one eagle and the second was five years ago so well it's not uh worth to to risk uh using it the most robust uh library is obviously tensorflow.js so i guess you know how tensorflow works we have the tensors and also and the flow the connections here that there's obviously the linear regression explained here uh so the graph and this is how you import the tensorflow.js uh and work on that so yeah if you have also the same for java go and other languages also javascript so the back actually the color here is same for all of the libraries so that's good because uh it's it all only depends on the on the low level api right that's it actually implementing what's what's in the kernel uh in this specific language so that's how how you use it in uh also as you can see because we have let also es6 that you can use for that all right uh here's the implementation of of the linear regression right easy implementation with some examples of a training set here's also said that the gradient gradient boosting right so optimizer sgd with some learning rate that's easy also to implement in in javascript um right when it comes to the cameras i mean i know that now keras and tensorflow is it's not the same we cannot tell it's the same but you know they cooperate well and you can use cameras and tensorflow and other way around uh so that's not really possible in in javascript i mean there is something like that called keras js but still i mean you can load the mod the the model like here there's a bin file with file so you can build something in keras and python and actually use it in javascript but building building models and cameras js it's not the most no it's not the thing you want to do on daily basis because it's not so convenient it's not so easy to do yeah some of them some of my details here clear what next do i have here um let's just go back to yeah that's the same with python right the data frames if you want to compare how to work you see the output is a bit e better some visualization visualization using the uh method lip or the one library that's actually used with data frames so you can see it's a bit easier to manipulate uh but still when when it comes to the to the ways how to visualize it's a bit easier to in javascript data sets also have a label in cycle learn that you can use directly to test your models like here the linear regression decision tree and obviously denser floor as an example in keras just to compare between these two all right so let me show you some images of here we go skin cancer that's now you can see three uh moles skin moles uh if i would ask you which of these what are what if let's say from the left you have mole number one number two and number three uh what which of uh the one here is the skin cancer so probably i don't know you would say i don't know one right or maybe i don't know maybe two uh because it's you know it has some this just kind of a different kind of net here or maybe the one on the right uh actually the number one and number three are uh uh that was confirmed by the histopathology tests so there are cancer cells because both i mean all three moles were cut off and check on the microscope to check if there are moles i mean if there are cancer cells so number one and number three are uh you don't know that but i know that that because i'm working on melanoma for about 15 years now i can see some patterns here that are specific for melanoma the one on the right here that's an obvious pattern you can see here it's kind of a structure that looks like gray blue gray blue white structure here this one here and this means that the advantage of the advanced i mean how advanced is the cancer is uh it's quite i mean it's it it it already reached the vessels right so it means it's uh it's an advanced stage of the cancer uh because of this color here so this means okay you need to cut it off and actually really do some serious serious uh um treatment just not to get uh other because you're not dying because of skin cancer directly but because of the what comes next so different kind of cancers in every every every part of your of your body so this is a this is our advanced cancer here on the lab you can see there are some stripes and some you see the dots here right the dots this this mole is also not regular it's asymmetric not regular the borders are not well i mean they're not smooth it's more sharpened uh so that that's also some does these are actually the patterns that are also specific for melanoma so the dots you can see it's going deep uh here the the stripes here right it's also something that is uh specific from this is a pattern that tells us oh that's a melanoma also the best is here around that's also a pattern to tell us oh that's something dangerous not always not always but that's uh that's something that should uh let's say star you should think about it if it's really something dangerous or not if there are some this kind of vessels here surrounding um so that the three four patterns i just mentioned here uh are specific for melanoma and actually it is a melanoma the one in the middle even the net inside it's it is not a melanoma um this one is quite it's not super symmetric but not so bad compared to this one or this one uh we have different colors and melanoma analysis we have six colors right so we have skin color light brown dark brown blue and and red so you can see here we have a few colors some nets but that's that's only suspicious so how the doctors do that uh how two doctors do that uh they use different kind of uh different kind of wavelengths of light uh or different kind of scarring methods i mean both actually in the typical scary metal like abcd or some point scoring methods of three points car scarring metal they use just a typical wavelength of life just a visible light so they only see the let's say the first levels escape in hunter's call they use different kind of wavelengths of light so you also see what's also visible in the other cells inside of the skin each of these use different kind of uh different kind of patterns like abcd stands for asymmetric border color and differential structure or diameter depending on that on what kind of paper you're looking for seven punch score is based on seven patterns overall we have more than 30 different patterns that are specific for melanoma so you can you can imagine for the medical doctor how difficult it is to really find to know all the patterns and find them on the picture here's an example where actually i imagine of images where different kind of wavelengths of light are used you can see here infrared you can see the cell the vessels here right both in both cases it's bad because you can see there are too many of them so many inside and here you can see a luck of vessels and here it's um this is cool again that you can see the car again in the skin and you can see some white dots here in in in this image that's also not good because this is how deep this this mole actually goes inside your body so what i what i did or why what many people do when doing the research there are two types of let's say researchers that are done one of it is actually using uh some image processing matters trying to extract the mode and find some some panels based on that like like here on the binary binary image or they just use some deep neural network to find the patterns or actually not pattern because patterns is that's something that will overcome what the what the researchers are not focused on on finding different illnesses like melanoma right and for example in the isic you have i guess seven or eight classes so seven or eight illnesses that you can define but this is only just a few but only illnesses not the patterns so the next step or the next stage would be to find the patterns not the illness itself to give a let's say to support the dermatologist to find oh there's a pattern that you need to take a look and that's that's why actually where the research is not now going uh here you can see if you take a look it's it's well it's not smooth i mean the border right but still it's not so bad how it could be so let me show you some examples here we go i guess it's uh here we go so border for example so what usual people do they remove the border right here's an example from an ise archive just to get only i mean because the images are like that so what you do to just cut the background uh it depends on what kind of thermal scope you use because some of them are flat like here so if you if you you know take it like that so it's it's strict uh connects with the skin then there is no no issue like like you can have here with the dark outside uh still there might be some light from the outside if it's if if you are looking on something that is not really on the flat if the skin is not totally flat like like i don't know if you put it i don't know here for example you cannot put it directly to the skin so there is a there is also some light coming from from the outside so that's the first challenge that you many many researchers approach or actually work on the other thing is about actually removing the hairs that's also very very um very um a big challenge now because it's not easy to remove the the hairs uh so extracting regions that are regions of interest more more actually images i know if you eat breakfast that's not the first thing that you want to see but this is how actually skin cancer look like in in in the zoom obviously so it's usually between 20 and 50 times so together with a polarized light you get the structure inside you can see here it's very small one uh and if you extract it you you can get actually something like that so we just take the most important part of the mold out of it so that's an exact step uh what you do next you can also calculate the sharpness or asymmetry of of the of the mole here you can see there are different ways how to do that i know it's written in python and you probably expected to have it in javascript but no tomorrow i developed in python but i use it using javascript so there's some binarization here and we just extract the border and try to find uh [Music] the border sharpness like like here right depending obviously on the on the parameters we set we can we can measure the border sharpness right so that's another thing uh yeah symmetry checking i didn't show that so what do you how usually how usually researchers i mean the dermatologists do they divide the image by into eight different regions and try to find the symmetries uh and actually get the numbers out of it how how how actually symmetric is the um the more or the blue veil for example that's the most typical uh pattern so um that's not i mean you can use it you can actually not you don't need to use any you know sophisticated neural network to find it out because it's that the color is very very very clear uh because you just extract the blue value in our case uh we have found the threshold uh here is actually we use also some from the relative for that but it's not something that you actually do need to do for for that because uh a finite triangle is enough to find the that's that that's this one that's that's uh here this color this is the blue veil now blue white whey veil blue gray white depends on the on the on the book you are reading about this this pattern all right um so what is also more very important is to uh als because i i why i'm saying about why i'm talking about uh the neural networks is that well finding cancer it's a medical topic and trying to build a model that is will be accepted by the fda you need to make it explainable what it means uh well if you have for example our typical neural network here we have one two three layers three layers here easy to train higher accuracy 98 with just a few epochs easily but if you try to explain that it might be hard because if you draw if you print the weights from one layer only it's something like that so trying to explain all of the weights it's not possible the deeper the network is the last possible it is to actually explain that so usually you can do what you do to make an explainable method i mean to build a model that is explainable is to use a white box model method like here that's a decision tree it can be easy drawn and actually it can be converted to some switch or if statements easy one or you can uh try to try to explain the neural network using for example uh the grad camps right that's that's another thing that you can do all right uh i think i'm running out of time so let me just go back shortly to the slides uh this is how actually the first first poc looked like so uh a patient uh example page take a photo and actually uh the the abc panels were actually checked for actually it was found then now because i cannot show you the the current version because it's actually will be used best by one of the german company it's uh it's much much more advantage because we show exactly where the pattern is and how it looks like so what are the pros uh using javascript easy for prototyping especially this kind of a solution where you have something some hardware that actually need to work with some with some mobile phone uh you have a support i mean there's a huge huge community of javascript community uh so if you have if you and actually javascript is not not a language that is hard to learn uh it's because of the i don't know react native solution they work with plenty of different models of i mean mobile phone models so you can easily move it to different kind of uh easily use it on different many many many uh phones also on android if you want to um so that's good and obviously can be easily moved to the web app but it's not so good uh well when it comes to javascript there are not so there are not such a good support when it comes to analog so actually moving into production it's just quite hard uh even the community in dallas from committee is huge um there is not such a such a big community when it comes to the machine learning topics related to javascript and then many of such of the libraries are absolutely deprecated or well not robust enough to really use it in production that's all when it comes to the presentation so i'm now ready to answer your question i see some questions already um here in the q a section um have you got okay uh one question here have you guys tried using cannabis reset to help meeting in this problem or how it he reverses you mean the cancer um healing the cancer with cannabis well i would recommend that i mean i'm not a medical doctor so i think it's a more like not a question to a medical doctor but i think i think uh well that's not the right way how you i mean i heard about canada's to be used to you know to um as a painkiller uh solution but not as a solution to beat cancer uh especially melanoma which is a very very progressive very very malignant cancer that that's actually even the the current uh i mean now it's much better than let's say 10 years ago uh but even if you if you if you actually take some drugs um many many drugs are just not not good enough to repeat this cancer that's why we have so many deaths uh when it comes to melanoma so uh i would recommend that but again i'm not a doctor to recommend the treatment i think uh there are better persons to to do that than me all right uh any more more questions why are the other questions i'm not seeing any others but maybe people are typing out so maybe i can talk about our next week's webinars and we can see if any more questions pop up and then um perfect so uh next week and maybe i should just share my screen um that might make things easier all right so next week we have two webinars um on may 31st it's between the spreadsheets classifying and fixing dirty data for data science with susan walsh who is the founder and managing director of the classification guru um so i mean we all know and carol can probably tell you this also uh real world data is never clean it's always there's always some issue with it so uh susan's gonna you know walk us through that and show us how to fix that if possible and then on june 1st at 11 a.m pacific so an hour before uh the previous day it's r and python the best of both worlds with and i am sorry bowie and um i'm going to butcher your last name to uh boeing angelov i think i hope that's how i pronounce his name um so he's the chief technology author officer at um i'm gonna say that's van or v triple a m um and he's just kind of taking us through um more than just the pros and cons of r and python both but he's saying you know the the argument between which language is better is over and he's going to take us through you know how do we use you know how to use r for this how to use python for that what are the differences between the two um and which is the best for your scenario and carol uses python so um yeah so that's what we have going on next week i hope we can i hope you can see that carol can you see that screen i can see your screen okay perfect um so i hope to see you all next week i'm not seeing any other questions so i think we can uh we can end here so thank you carol so much um we'll have your recording up and available um either later today or early tomorrow and i will make sure to send that to everyone as well thank you everyone for joining whether it was in zoom or on our live stream we really appreciate having you and um if you're interested in any of our data science programs we do have an information session tomorrow i believe it's at 9 00 a.m feel free to drop in um and learn about learn about what we do at data science dojo um if i don't see there i hope to see you at our webinars next week so uh thank you carol thank you everyone and i hope you all have a great rest of your day thank you very much thank you bye