Introducing Zero Shot Object Tracking

Key Takeaways

hey there this is jacob from roboflow here today to talk about zero shot object tracking so zero shot object tracking is basically object tracking with an object detection model where we don't have to train any separate classifiers in order to do object tracking so let's zoom back here real quick what is object tracking object tracking is when you use an object detection model to make predictions and those predictions have a sense of continuity across frame over frame so here you can see this car here the silver car is identified as number one the number one car for all of the frames where the object detection model is is running so that requires an extra step from frame over frame to be matching up which frames recognize in previous frames so previously in object tracking uh what was done was a classifier was trained across the whole stream of object tracks this would create uh object track features which could then be compared within regions to identify which track is most likely uh to be kind of continuing the continuity of the object now the annoying thing about this was that you would have to train a whole another model a whole another classifier and you'd have to annotate these different streams of objects and now that is something that is rather unfortunate from our point of view so we decided to try to work and strip all of that out by using the openai generalized clip model which is a model that basically is a zero shot object a zero shot image classifier so you can use the features from the clip model the zero shot features and do the whole same object tracking routine it's crazy and you might think well this is some uh something that just lives in research and maybe this is something that's just living inside jacob's head that's not true it is open source and we have it open source available for you now here at the roboflow dash ai backslash zero shot object tracking repo we're really excited for you guys to get a hold of this and start working on this one of rovlo's first open source projects so if you want to use this uh basically all you need to do is have an object detection model ready to go and then you can hook that into this repo we'll download clip which will be doing those object track features uh you download some requirements um and then you just simply run this clip object tracker dot pi with the source of your video and then the url uh where your model is hosted on uh roboflow with roboflow train and uh your api key here now of course this is showing how to do with the realflow inference api but we will also be coming out with ways how to do this with yellow before yellow v5 other object detection models that you might have trained and of course uh prs are always welcome on this repo and always feel free to make issues and we'll be uh taking a look at it and building it uh over the coming weeks and and months so that's a quick tour of the zero shot object tracking with rebel flow of course i encourage you to dive in a little bit more and to try this of course to try this on your own data set with your own models because we all know that that's where things really start to get useful in computer vision these days so uh to sum it up this is a zero shot object tracker here trained um this this predictions on these cards are done using uh public cards data set uh on revlo universe which you can adapt with your own data sets and uh there's no extra model that is being trained here it is only the object detection model that is trained uh on the cards and then you're able to then do object tracking which is a really cool thing uh hopefully you uh get some great use out of it and if you enjoyed this video please like and subscribe please check out our repository linked below and we'll see you in the next video