Transfer Learning in Computer Vision | What, How, Why

Key Takeaways

let's talk about transfer learning in machine learning transfer learning is when you take what another model has learned about a similar domain and used that to train your new model from a predefined starting point so let's take an example that's not in the context of machine learning okay let's say you have two friends and you're trying to teach those friends how to skateboard we'll say one friend has never skateboarded before never snowboarded never done any sort of similar activity whereas another friend already knows how to surf okay now which of these friends do you think is going to pick up skateboarding faster probably the one that has some experience in a related domain the one that surfed before in a sense this is what transfer learning is in machine learning it says let's take what another model has already learned that's similar to what i'm trying to teach my model that's new and start from there when we do our training i'm joseph from roboflow and today i'm going to take you through how you can use transfer learning in computer vision problems so that your object detection models can improve faster okay so the example that i'm going to be working with today is a chess piece data set okay so this data set is available in its full form on public.roboflow.com i'll put the link in the description what the dataset is is it's actually a series of about uh well this one has 189 images and 1762 annotations across 12 classes each of the chess pieces and the goal of what we're trying to do is teach our model to do object detection of each of these individual chess pieces okay seems simple enough so what we may want to do is we can start training from totally from scratch right that means that we take a model where the model knows nothing the weights are randomly initiated and then those weights learn over time and try to get to the closest point possible we could also start training from a different checkpoint now that other checkpoint we would hope would be something that's somewhat similar remember my example with our friends if we have one friend that we're trying to teach them both skateboarding and we have a friend that has learned how to snowboard maybe having that friend start would do much better in the lessons whereas the friend that's never done anything in the related domain will take longer to learn that's kind of what we want to do here so i've actually gone ahead already and with my data set of 189 images i've generated two versions now these versions don't have any augmentation the only thing that's been done is their images have been resized to 416 by 416. okay so that's a 416x416 image and same with this one okay now when we kick off training we're going to use roboflow specifically roboflow train to kick off a training job now we're going to do two different training jobs okay to demonstrate the power of transfer learning in roboflow when you click use roboflowtrain you're basically saying please roboflow train a model for me without me needing to write any code and give me the results of that model in the form of an api or a model i can use for future labeling or just to benchmark my existing performance now you'll notice that i can do two things i can train from scratch or i can start from a checkpoint train from scratch means don't use any pre-existing weights that is the rates will be rent the weights will be randomly initiated and the model will just learn the best that it can alternatively i can start from a checkpoint now logically this checkpoint that i start from should be something that's a similar domain now in roboflow those things that i start from could be data sets that i already have on my account so i actually already have a different trained model on my account of chess pieces from this example here or i could start from a public model and so the public models are models that are available on roboflow so for example the common objects in context data set the coco data set is where i could start my training from now i'm going to do two tests here i'm going to do one model where i kick it off from the coco data set and then i'm gonna do another model where i kick it off completely from scratch okay now if you're unfamiliar with the coco data set it's also available on roboflow public and basically what it is is a data set of hundreds of thousands of images of objects in common contexts things like forks and spoons and persons and cars and dogs and microwaves and things like that and so what i'm thinking here is if i start my training on the chess piece data set i'll try one from the coco data set to see if the common objects in context data set approximates pretty well and then i'll do another one completely from scratch so i'm going to start training from the coco data set and that goes ahead and that training job starts and then on the other one on v2 i'm going to start training completely from scratch okay so those two have kicked off now while those start to train let's talk through the advantages of transfer learning there's a few things that transfer learning does for us one it may and likely will improve the accuracy of our model because think about it if we're saying that our model needs to learn but it can start from what another model has learned in a similar space then that model may optimize to learn the new domain even better so in other words the first advantage is it may make for more accurate models the second advantage is it actually will maybe likely will make our training time be faster if we have a model that's starting from a predefined sp checkpoint of a similar domain then we would think that the new model can learn you know the rest of the way whereas the model that starts completely from scratch might take a little bit longer to optimize its loss function and understand our new domain more successfully again think back to our example of teaching your friends a new skill if a friend has experience in a similar domain you would expect they can pick up the new skill faster than if they didn't now transfer learning isn't totally a free lunch mind you um and we'll talk more about that after we let these models train so i'm going to let these models kick off and train now uh while those models train there is one other thing that i do want to walk through so we described transfer learning as being advantageous if we were starting from you know maybe a related domain but we could even do transfer learning on the same domain so what do i mean by that let's say that with my chess pieces data set here you know right now i only have 189 images now let's say that in theory i had gone out and i'd collected and i labeled additional images i actually have gone ahead and done that so i'm going to go here and i'm going to click add more images and then i have actually gone and i've created i'm going to select my folder and on my desktop here i have an additional 100 images and annotations so i'm going to select that folder and i'm going to click upload yep and all of those images and their annotations are going to be processed here for my data set and as i said i have an additional 100 images now why is this advantageous now when i train again in the future i don't want to start completely from scratch right if my model has learned what 189 images and 1700 annotations look like then i want to start from that point in time to add an additional 100 images so these additional 100 images i'm going to add them to my data set and i'm going to add them all to my training set in fact so they'll all be added to my training set and those are all going to be uploaded now in theory right if i did this out in the wild and i've collected new images of the same domain when i start training again i don't want to throw out everything that my model learned before i collected new images i want to start from the same point where i last left off so with all these new images added to our data set we can go ahead and create a new version and then we can kick off training again but but we'll kick off training from a checkpoint and that checkpoint will be the ones that we just started so i'm going to wait for these models to finish training and for this upload to finish and then i'll come right back and i'll report the results of accuracy and the time that each of them took to train so hold that thought i'll be right back and we'll do a little movie magic okay so our 100 new images finished uploading and both of our models finished training which means we can compare the model results v1 versus v2 and then we can also look and see if we start with our training checkpoint for the next 100 images how that's going to change our model performance so first and foremost remember model v1 the dataset v1 was the one that we did the cocoa weights pre-training checkpoint on this is the one where we did transfer learning v2 was the model where we started from scratch where the weight started in a random position and learned only on this data set so if we look at the v1 model results we see that it does quite well the map here is 98.5 the precision is 73.3 and the recall is 99.3 now i'll put a link in the description for explaining what each of these metrics are in greater depth if that's of interest don't forget to like and subscribe to our channel as we post additional updates like that but in general here we're going to be focusing on map so 98.5 for the transfer learning model now before i tune over to the v2 let me also tell you something about the transfer learning model we kicked off both of these training jobs at the same time but the transfer learning model finished 49 minutes before the model that started training from scratch so the v1 model took about an hour to train and the v2 model took about two hours to train from scratch okay so we already know the transfer learning model was a little bit faster now its accuracy is quite good how does that compare the v2 model we see the map is 97.9 precision 53.5 and recall 99.6 so the mean average precision of the transfer learning model is ever so slightly higher at 98.5 the precision is much higher at 73.3 and the recall is actually slightly lower 0.03 lower um at 99.3 that could be attributed to noise or it could be actually that it learned something very specific about our chess pieces um but in general here we see that transfer learning has done a good job for us our model trained faster and it's ever so slightly more accurate now okay i told you that i added a hundred new images to this data set you saw me upload them so now we have 289 images and that translates to having 289 images in 2870 annotations so over a thousand more annotations now i'm going to actually go back to our data set here and i'm going to generate a version i'm just going to call this version v3 and this dataset version is going to have the same pre-processing step so it's going to be a 416x416 image that we passed to our model and no augmentation so it's just going to be the raw images that we're going to pass to our model here and so this v3 one we're also going to kick off a training job so we'll click use roboflow train and remember we can train from scratch or we can start from a checkpoint except this time the checkpoint we can start from is we can grab a checkpoint from the chess pieces data set and we can actually grab a model from one of our existing ones on our account so i already have the as i mentioned that chess pieces data set on my account so i'm going to start training from that exact model and i'm going to kick off training and while that kicks off we're going to go ahead here and use a little bit of movie magic and wait for that to finish while training is in progress and we're going to come back and update you with how long that took in comparison and we'll see the metrics and results to see if there's any notable difference okay so see you in a bit the model finished training okay so let's recap what we did we first uploaded 189 images we did training from a checkpoint on the coco dataset we also did training from scratch then we added 100 more images to our data set and we started training from a checkpoint on that same model so our model enough to start from scratch it could remember what it learned from those first 189 images and then apply that forward on our next 100 images so that was v3 i need a little drumroll the results 98.9 percent map so remember v1 saw 98.5 v2 so 97.9 that was training from scratch and v3 saw a 98.9 percent map now notice the precision is slightly lower and the recall is a little bit higher i mean those numbers kind of depend on the thresholds that you use i'll link in the description more about those metrics but all in all we've done it we've successfully used transfer learning in computer vision please like and subscribe to our youtube channel to get up-to-date updates about videos tutorials blogs and new features in roboflow and thanks so much for tuning in happy transfer learning