A Guide to CoreML on iOS

Key Takeaways

hello world it's Suraj and this is a guide to Kor ml that is Apple's new machine learning framework for all of its iOS devices we're gonna go to the development workflow of what it's like to work with core ml we're gonna talk about its API we're gonna talk about its features we're gonna talk about its pros we're gonna talk about its cons and then we're gonna build a spam classifier for text messages on iOS on an iPhone device to really implement what we've just learned and so that's what we're gonna do today I'm very excited for this because Hormel has a lot of hype right but there's it's not very well understood in terms of what's happening under the hood so let's talk about that right so Apple released this framework this year at WWDC with a lot of hype has always got to be hype when Apple does anything right and so actually let's start off with the demo right let's start off with a demo so I'll type in a text message and then I'll classify it as ham or spam ham means not spam right that's it that's what we're gonna do just that simple thing okay that's gonna be our demo so that's what we're gonna do it's gonna classify it as ham or spam and we're gonna build that using the messages framework which which allows us to build a custom app that uses iMessage or we could build an extension to the existing iMessage app which is what we'll do okay so that's what we're gonna do today and okay before we get into the features let's just talk about some history here this is not the first machine learning framework that Apple has released in fact last year it released two of them - not much not as much hype as core ml had so get ready for some acronyms the first acronym is MPs CNN which stands for metal performance shader convolutional neural network and the second one is BN NS which stands for basic neural network subroutines actually it's BNs so there's always more acronyms there but both of these that let developers build convolutional neural networks both of them MPs CNN was for the GPU and then B n NS was for the CPU that was the difference there and both of them let you run inference on your device so remember there's a different there's a difference between training and inference none of these frameworks are gonna let you train your model on your iOS device but they will let you run inference that means running a pre trained model on your device without needing an internet connection so both of them were released last year they released a set of data types and function types and different types of layers and for all intents and purposes that let you build cnn's but that was it it was also very hard to create a dot ml to create a model file a lot of developers were having trouble with that so based on the troubles that developers were having and just the fact that machine learning is happening everywhere and Apple is seems to be just now catching on if they built core ml so what core ml is also I want to say one more thing so the reason they built two different frameworks one for the CPU and one for the GPU is because the CPU is sometimes faster than the GPU for inference not for training for training the GPU is always faster so you kind of have to guess and check with using both but what Apple did was they said well we don't have to we don't want to have you have to worry about all these things so we'll build a framework around both of them so that runs on the CPU and the GPU so that's what core ml is Hormel is built on top of the previously to release libraries and acts as a layer of abstraction on top of them and when I say abstraction I mean abstraction seriously it is almost too easy to use like you'll see what I'm saying it's very easy to use and then on top of core ml they built three other api's one is for one is for one it's called vision and it's for all of your image analysis tasks object recognition classification and then there's foundations which is for natural language processing that's for sentiment analysis predicting the next word in a sequence sequence all sorts of text related machine learning and then there's gameplay kit and it allows your app to evaluate decision trees so those are the three domains that build on top of core ml and so the reason they built all three of these on top of quorum L is for modularity right so let's say you have an app and what you want to do is classify where somebody is so part of your app is tagging everything you see so let's say you're on a beach it would tag all the things that that the app sees it would tag the water it would tag sand it would tag sunshine maybe some beach volleyball players and then once you have all those tags using the vision library then you can use the NLP library foundations to then take those words and classify what scene it is and it would say based on these words this must be a beach and then you'd output a beach so that's just one example of how these libraries can interact together and so for the development workflow what Apple has done is it's essentially built a PDF for machine learning models right the PDF is a standard format right and so that's what they try to do with machine learning models it's in the dot ml model format and so you can use the dot ml model on any iOS device Mac Mac Apple TV iOS or iPhones all of it and iPad all of it and so it's a very simple process it's a two-step process the first step is to load a pre-trained NL model the second step is to make predictions with it right the training isn't happening on the device so it's that simple so but sometimes and so when I say using a pre trained model Apple has a list of these pre trained ml models that I have a link to in this Jupiter notebook and they're they're all these very popular models mobile net squeeze nets googling Nets ResNet these are all convolutional networks by the way inceptions huge vgg all of their pre trained models are convolutional networks and so you can just use those but let's say that you don't want to use their there are pre-trained models let's say you want to build your own then it becomes a three-step process the first step then becomes to convert your pre trained model in whatever machine learning library you're using whether it's cafe or scikit-learn into a dot ml model using a tool that they've created for this called core ml tool it's a it's a Python library we'll talk about that and then you do the next two steps load the model and make the predictions and so here is an amazing list of all sorts of machine learning models that you can use for core ml right there's a bunch of pre-trained core ml models and then they have models for different libraries that you could then convert to Cornell now I had do have a very disappointing things to say they don't have support for tensorflow I know what are you there is no tensorflow it's okay whatever whatever whatever you can build your own conversion script for your tension flow models to run encore ml in fact there must be something on github I'm just making a prediction right now prediction anyway so it's either a two-step or a three-step process and so it's the ml model is like the PDF of machine learning all you have to do is once you've trained it you drag and drop it into Xcode so you just drag and drop the dot ml model and then you drag and drop your data set or whatever you want to test your model on and then you can call it very very simply I'll show you the code right here here's a code the sample code but basically when you import the model into Xcode it's gonna look like this you've got it's it's gonna parse the data inside of it it's a very it's very neatly packed for Xcode to read so it'll be able to say oh here are your inputs here are your expected inputs and their data types and it's description here's the output it'll tell you the learned parameters and the weights that means the weights and the biases as well so it's a very clean format for Xcode and so once you've imported that model you can call it just like this you can call it by saying let model equal ResNet 50 let's say for example that's the one we're using and then you'd say here's my image let pixel buffer up type CV pixel buffer equal my image wherever it's located and then I'll just make a prediction using model dot prediction feed it the image as the parameter and then print it out right that's it that's your prediction it's that simple so let's talk about the pros and cons right so here's an image of all of the machine learning libraries that are compatible with core ml that means that that means that the core ml tools Python package can convert a model trained with these frameworks into the dot ml model format that you can then run that you can then use to run inference on your device caffeic arrows XG boosts live has SVM scikit-learn Tory right so the pros are it's optimized for on device performance which minimize minim memory footprint and power consumption that's the one thing you know about apples library as opposed to any third party framework you know that it is gonna be optimized AF for its devices because they make the hardware and the software we also know that Apple really cares about user privacy that means use the privacy of users data so you that means you don't have to send the users data to the server it stays local and encrypted and it means that you can run and because inference is happening on device you don't need an internet connection so they could be in like a ditch or a prison or why am I think of these weird places but some are without internet connection and then it will be able to perform inference there right you don't need an internet connection and it decides itself remember it's a it's a layer of abstraction on top of the previous two machine learning libraries it decides itself whether to run the CPU or the GPU or both so it's it's it's self optimizes what it's running on and because it can run on the CPU you can run it on the simulator which you can't if it were to just run on the GPU because the simulator doesn't support the GPU yet and it supports many model types that's the last Pro it supports support vector machines they're all listed here it supports three types of neural networks a convolutional network a recurrent network for sequences and then a feed-forward network right and then you've got tree ensembles as tree ensembles like random forests and boosted trees linear regression and logistic regression okay so that's it for the pros and we have to talk about the cons right it's not perfect the first is that there is no there is only native support for supervised machine learning models that means that models that require labels there is no support for unsupervised models or for reinforcement learning so that's that that can be a big a pain and no training on device right so you can only perform inference on a device also it only supports certain layer types so you can just create a new layer type yourself and add it to core ml it's it's impossible to extend core ml's of native layer types and it only supports a specific set of training tools not tensorflow however you can write your own custom conversion script for tensorflow models you can't look at the output produced by the intermediate layers only the output layers it only supports regression and classification that means no none of the unsupervised techniques like clustering or ranking or dimensionality reduction and my biggest gripe is this kind of ties into no training on device but no federated learning right federated learning is a technique Google actually published a blog post on this very recently but basically you can utilize all the phones that you deploy your app to you can train it on their data locally instead of having it train on a server and so you can combine all of the that training into one big model that is then deployed to everybody so every you're you're learning from all the devices that you're that you're deployed to and so killed you are core ml is super simple we saw a very simple code simple already but it's limited in its functionality so if you want full control you're gonna have to DIY with either the two native libraries that it's built on top of or you can just use another third-party framework or ml is not the only way to do machine learning on iOS devices there are other ways surprisingly right and I've got a list of them right here there are a lot of third-party frameworks that work with iOS that leave you a bunch of different machine learning tasks so if core ml doesn't fit your needs specifically or you don't want to extend it then go ahead and use these alright so what are steps in this tutorial art are steps are going to be first in Python to look at what it means to import the data set train a model and then convert that trained model into a dot ml model then we'll go into Xcode and then in Swift we'll drag and drop our data sets are trained ml model into Xcode then we'll write our basic prediction code and then we'll run the app alright let's get started with this so for our Python code so I'll write out the swift code but we'll just glance over the Python code so for our dependencies for the Python code we have to write we have to one is for numpy for matrix math and then the second is scikit-learn in all of its sub modules right we're gonna run we're gonna train three models and then we're gonna pick the best one and I'll talk about what those three models are but you can get a little hints from here as well as two techniques to vectorize the each of these models so in total they'll be six different pipelines that will build so for each of the models we'll try out to different vectorization techniques right so two two and two so which makes six right so that's it for scikit-learn for numpy and so core ml tools the the Python package only supports Python too so if you if you have Python three you can use these commands to initialize your Python environment and then you can import core ml tools right just like that so I'll go back up here compile but uh okay and so now we can import our data so let's take a look at our data set really quickly here what does our dataset look like we have an SMS spam collection data set and it's a bunch of human labelled SMS messages either ham or spam right spam or not spam alright so that's the data set we're going to use and so what we can do is we can open that data data set it's a txt file well go through it we'll split all the lines and then we'll convert it into a training and a testing set using scikit-learn strain test split function we can print out the training set to just see you know what it looks like it looks kind of messy but these are some text messages and it's just got a label ham or spam that's it right binary classification right so that's it for importing our data and now we can look at our models right so we're gonna use three models and each of these models I've created a video just explaining the entire model so definitely check out each of these videos in my math of intelligence playlist I've got one for the multinomial naive bayes support vector machine and for the random forests which are the three models that we're gonna use but to just go over them at a high level multinomial naive bayes is a specialized version of the naive bayes classifier these are all classifiers where it's just computing conditional probabilities for all the words in a set of documents right it's just iteratively computing conditional probabilities but definitely check out my naive Bayes video on that and then I have we're gonna use a support vector machine so it's a little refresher here writes the support vector machine is a type of machine learning model that can classify two different classes and what it does is it builds a hyperplane by using the by using these support vectors that are the points that are the closest to each other between the two class and then maximizes the margin between them and draws a hyperplane right in between them and so when we add a class depending on what side of the line it goes on we can classify it and for a random forest a random forest is a set of decision trees you know yes no it just it asks a series of questions and then it the result is what it what something is it'll classify it and what we can do is we can say let's create a bunch of decision trees have them all classify some data have them vote and then pick them pick the winner right what is the most likely class based on a set of decision trees random forest right school that's why it's called a forest because this set of trees and then in terms of vectorization strategies we're gonna use either the count vectorizer which is just basically a bag of words it just counts the number of times a word appears or we could use a more advanced way of doing this called tf-idf and what this is is it's a technique to score each vector right for a term I in a document J so for each of the words what we can do is we can say the score it's tf-idf score is the number of occurrences of I in J times the log of the total number of documents or SMS messages in our case over the number of documents containing hi and so we'll compute both of these the count vectorizer and the tf-idf for all three of these models giving us six different pipelines to use right so here's here's how it's building the pipeline right we have six different pipelines for each of the models we'll try out both tf-idf and count vectorizer once we have all those pipelines we'll put them into a list and then we'll perform classification using all of them and so we'll find this is running right now or running live will find that the support vector machine using tf-idf wins right this has the highest accuracy 98 98 98 so that's the one that we're going to convert into a ml model file so what we'll do is we'll say ok we'll get all the ordering of the words and from this dot txt file and then we'll vectorize it and then we'll convert that model what we'll say here's the model the linear SVC fit it to the vectorized data and then use core ml tools to then convert for specifically for scikit-learn it's got converters scikit-learn dot convert the model based on the two rows the message and the label and that will create a core ml model in memory and then we can save it using the save function as a dot ml model file so when we when we compile this it's gonna save it and then we can then drag and drop it into Xcode okay so now that's the first part we did it we've trained a model in Python on our machine now we're going to deploy it to Xcode so let's check out Xcode now so in Xcode what I what I have here is it's is the a skeleton file it's a skeleton file all it has is it's got a storyboard with you know label and then a button right and the the button and the label are wired back to the view controller right this is in swift we've got one view controller with all the logic which is no logic as of now and then we have the storyboard right so what's happening here is that for the view controller we have an IB action that's gonna fire every time the button is pushed so inside of here we'll put our prediction logic and then we have this other function for it to perform tf-idf that we're gonna fill out okay so let's let's do that but first let's drag and drop our created ml model to our project just like that we'll drag and drop it and also our spam collection our data set because we can test it out on that on the training data as well okay so then let's go back here and let's start writing this out let's see what this looks like okay so so first of all let's retrieve the text that the user has typed in where we want to retrieve that message so then we can classify it as spam or not spam and so what we'll do is we'll say if the text is copied let's go ahead and vectorize that text by saying let vector equal tf-idf and now we haven't actually written out the functionality for this function yet but we can just say well if we feed the text into tf-idf it'll create a vector for us and then in a do statement make the prediction right we'll say let the prediction equal try the message classifier right that's our message classifier make the prediction the message classifier is named right there so we can just call it just like that and then we'll feed it as the parameter the vector that we computed using tf-idf and that's going to store the prediction we can print we can print out the prediction for our own logging purposes and then we can - the what we can send back that data to the interface or the storyboard by saying set the label to the prediction and then so we'll also have a catch statement so if it can't make a prediction then we'll set the label to no prediction just like that and that's it for our logic right here and then we'll go to our TF D F function and so now let's write out this DF DF code so what's happening here is we can say first we want to import the words ordering and the words so we'll go ahead and import both of those just like this and so it's it's finding where both of these text files are and the next step for us is going to say is going to be let's create a do statement and the next step for us is going to is going to be let me just write this out like this just like that and so the next step for us let me just paste this in right here boom like that or delete this okay just like that and so we're gonna say let's retrieve the ordering so so inside of our do statement we'll retrieve the ordering data we'll retrieve the SMS data we'll remove the trailing newline from both of them as well right removing the trailing newline and so once we've retrieved both of those text files we're going to vectorize our words just by using this we're gonna vectorize our words we're gonna have a collection of each of those words that we're going to split by the separator the empty space and then we will vectorize each of those words and so now for the vectorization part here's where the real meat of it it goes right so we're gonna say for each word let's count the number of times the word shows up in an SMS all right we're going to count the number of times it shows up in an SMS and then we're going to multiply it by the log of the total SMS messages divided by the SMS messages that contain that word and so that's going to give us two tf-idf score which we can then store in the vectorized array and then we'll return that array of all the tf-idf scores of each of the words in the SMS documents and that's it for that and then we can go ahead and compile this and it works just like that so overall I think this is a very fun library it's gonna it's a great way to introduce regular developers who've never done any kind of data science or machine learning to machine learning because it's so simple to use if you've never coded for iOS before I would highly recommend it it's a lot of fun and if core ml doesn't give you what you want you can always use the two frameworks that it's built on or you could just use a third-party library all right that's it please subscribe for more programming videos and for now I've got to be thankful for automatic reference counting so thanks for watching