YOLOv12 Object Detection Training Tutorial

Key Takeaways

let's say that you're operating in model yard and you want to inventory all of the containers in your yard but in order to do that we need to find sh model that's capable of say identifying containers identifying where the IDS are so we can send them to CR now I'm going to walk through the notbook but I will mention that you can train your V12 models in rlow so here I'm going to run on this example image which I'll show and already our work for done this output two here corresponds with the CSV format today we'll be chatting about how to find tune Yol V2 uh Yol V2 was released last week by an independent team of researchers and here we are now being able to find you the model on an industrial use case so for this uh kind of next 20 minutes we'll be talking about what Yol V2 is how to fine-tune the model rlow Yol V12 support and finally how to build a vision application with the model toward the end we'll also have time for Q&A the Yol V2 is introduced in a paper Yol V2 attention Centric realtime object detectors uh primarily focused on object detection it was released by a team of researchers from the University of Buffalo City as well as the University of the Chinese Academy of Sciences H the model does have support for other tasks like segmentation detection although those are secondary and came after its integration into the tics python package um on the right here you can see an example of the model uh that we'll be training today running identifying IDs logos chassis IDs um and other kind of related classes for shipping container identification uh so you may be wondering like how does y V12 perform uh this chart directly from the paper and GitHub repo um shows kind of uh performance on two different axes on the left you can see latency um lore is better and then on the right you can see the me average possession higher is better and the way to reart like this is basically the closer the line is to the top left the better so on the left we can see that latency is lower across model sizes for y V2 when compared to its predecessors and similarly mean average Precision is higher when compared to his predecessors um with that s the performance benefits are somewhat marginal compared to previous generations so if you're comparing y v 12 Nano to yv 11 Nano that performance game that you're getting 1.2% with that said though if you've already got a great data set that you've been iterating on for several months maybe that architectur Improvement is exactly what you need to get that kind of last mile of performance out of your system and in addition to this kind of performance boosts it's notable that the model introduces attention to the backbone of the YOLO model um of course attention being part of Transformers so there's a lot of excitement around these architecture developments and seeing it combined with folo um only kind of adds to the impression that this series is continuing to iterate over time and learn from kind of the latest in Nal language processing so over the next a little while we're going to be functioning a Y2 model um as we would with any uh Machine Vision project we're going to start with a data set with that has been labeled H we're then going to install dependencies we'll start a training job we'll view our model metrics H then we'll test the model on images from our custom data set uh for this guides we've prepared a data set that you can use to follow along you can get that row.com container demo and and with that we're gonna get into find chuno so we also have a notebook at roof.com notebooks free to follow along with this notebook is at the top of our list fing V2 it walks through another example but you can just switch out the data set with any one that you want whether labeled in rer FL whether one you've already got uh on kind of on the back burner ready so let's kind of visualize what we're building here and let's say that you're operating in your model yard and you want toy inventory all of the containers in your yard you can do so with Machine Vision we've got all the pieces we can do OCR to get uh characters and containers so we can do container counting but in order to do that we need to find Tre model that's capable of say identifying containers identifying where the IDS are so we can send them to tocr so this model um that we'll be building uses a data set that identifies shipping containers the logos uh and the container IDs as well as go toide the chassis IDs too which are at the bottom um and this is really all we would need for that system the DAT set that we're working with today which is available on Rob FL universe and this is the the link that I mentioned earlier um it contains 150 images split across train test and Val and certainly for production use case you'd want to get that up more but um this is kind of a good starting point for us now I'm going to walk through the notebook but I will mention that you can train your V2 models in rubble flow we got support for this within uh less than 48 hours after the model was released which we're really excited about and we're already seeing people train these models when you're in Rob FL if you hit versions and then create new version after you've got that version you can hit the custom train button and Windows kind of pop up that uh mentions our Flagship model the rlow 3.0 as well as below that YOLO V12 and when you hit that and click continue h you can choose your model size then you can kick start a training job when you do so um K of machines will spin up and your model will start training and then you'll be able to deploy it on Robo flow with that say though uh we do have this notbook that um that I want to walk through this notbook um available uh on GitHub ready to use kind of gives a bit of background about some more of the technical details of Yol V12 and uh the kind of learnings from uh the attention mechanisms that have been implemented in the model and again those charts of um of the improved performance but whenever you're following these notebooks uh we start with a robo FL API key you can put this in your collab environments um and uh this will allow you to download any data set from rlow whether any on Universe which has over 100,000 data sets and that's where this data set is hosted or any in your um private account when you've set up your API key when you're training Yol V2 please make sure you do so on a GPU and the package uses flash attention uh we recommend training on an L4 or T4 and and like on an L4 for 100 XX of Nano we've seen it take like uh 5 to seven minutes for training of course as you H kind of look at larger model size is it's going to H take longer um so yeah run nid make sure your GPU is all set up before you get started um our notebook builds U V12 from Source um using the official GitHub uh repository for the package in addition we installed uh kind of our rubber flow dependencies we'll be using and the Flash attention dependency make sure that you've got that installed and when you are uh when you're following this notebook um otherwise you'll see an error um next we kind of like to do a demo with inference for this gu we'll be using supervision which is our open source package for processing detections from Machine Vision models um and uh we load with the ultral lithics package which That official GitHub propulsor uses and downstream uh and we load uh the PIP Arch weights for YOLO V2 large and then like like if you've uh used the old models before just passing image through that model call and uh we'll load it into supervision and this gives us access to supervisions vast Suite of things um model valuation and image annotation and more um here we can see yeah the model uh Bas way train of Microsoft Coco um kind of Trends in the right direction identifies all the common objects and this image of on for machine learning engineers and his dog um but the the thing that we're here for is fine tuning a model for your management I've gone ahead and run this just before um we got started downloading that yard management data set when you download the data set um in collab we have this uh these five lines of codes that make a couple different changes to the past in the DAT set yaml file and to make sure it's ready for use so make sure you run that then when you do you'll have the shaml file that can be pared by lv12 um so yeah you V2 we're loading the Y V12 small um for this demo um and we'll be using the ultra liic package when you're training uh just for this demo for 100 EPO it took four minutes we do recommend 200 maybe a bit more and to get a kind of better performance and then if you train on rubber FL it's going to train until your model kind of um hits uh hits it potential uh so you'll see uh training for more epic and rub FL potentially when you run this H you'll see kind of the the standard output and it'll go through the EPO and that'll take a few minutes and when it's ready you get the result from your model now because it is using um kind Vol allytics we have all of these kind of uh Graphics pre-calculated that show how our models doing and for only 100 EPO our models doing relatively well across these classes that's a Chassy IDE that Ide at the bottom of a container the container itself and the uh IDs horizontal and vertical as well as the logos there's definitely room for improvement and this is where um one we look at training for more epics like 200 300s and then also uh potentially using some more R's um model evaluation tools to be like okay given um this test set given this model uh where are potential inconsistencies they labeling uh that could be addressed to improve the quality of our model and with that say though looking at TR training graphs uh for this shipping container res that everything is trending in the right direction losses going down consistently epic after epic and which is what we want um now supervision um we used kind of the annotator API to see those bounding boxes earlier we also have an API for calculating mean average precision as well as several other metrics that you may be interested in this works by loading a DAT set from YOLO um we uh train using the YOLO V P txt format which you can import uh data sets um into rlow that use that format and export data sets from rlow in that um format we use uh that kind of static folder load it into memory in the detection data set and then we can pass uh each image from our data set and the kind of ground truth annotations um as well as uh the results from running that image through the model that we just trained the uh training job the best pytorch file or best PT file and we can use that all to calculate mean average Precision on our data set um so for this printing out the results and like we got 0.77 for me of 50 um which is good for 100 EPO um and and kind of just getting started with our first model we can also plot that across several different um AES this includes like uh the sizes of uh detected do objects so if you're working on a use case where you're detecting more small objects you may care about these small numbers um or large objects uh similarly so this gives you a nice view into how are your V2 bottels doing but the fun part is actually running inference um so again load the model we're going to load that data set um but this time loading from the test set um and with that kind of looed test set we can run inference on an image so we like to just take a random image from that test set and plot it this being an example and our models successfully got logos it got the chassis IDs it's getting these IDs from the sides um and um yeah everything is working like and this on another image it's going to cycle through um and here getting the logos it's getting the IDS even though that one's very slim uh there and and this logo could be a bit bigger but certainly localized the the region um and here in another image it's got more containers it's getting these front ones um and seeing these kind of blurry containers in the backgrounds that do that are present that be like looking at our data set quality to make sure that we've got labels for these kind of further away ones but certainly we've got enough to get started with um with our kind of inter modal yards inventory use kits so if I go back here um I'm going to talk about bit about y V2 in rlow so as I mentioned earlier you can import data um in any format or any common object detection format like Pascal fo XML or coco Json or YOLO data into rlow and then you can convert it into that Yol V12 format which is the same one as Yol v p txt any data that you label in robotow of course can also be converted into that format when you have your label data you can train models in our platform you can export your data and train them on your own too and then finally you can deploy your Yol V2 models now when you're working uh within rlow you can deploy your models with our serverless API which scales infinitely so um if you're kind of dealing with real-time applications or applications with a bit more of like a burst type um usage where maybe you use a lot of API within six hours of the day and don't use it as much uh our infrastructure is going to scale up and down uh as necessary you can also depl models with Robo flow workflows which I'll talk through in a minute and as always on your own Hardware with Robo flow inference um I will mention that uh when you deploy with roof flat um you are covered under our licensing agreements um and so uh if you are a pay R FL customer you're able to um use uh Yol V12 commercially per our terms and you can learn more about that rob.com licensing and what that we're going to build an application and workflows so workflows is our like a visual AI web application based Builder and using this tool you can drag and drop blocks to build compact station applications and the blocks that we support are kind of modular components of an application that when chain together create a full visual AI system so these blocks range from using models train rlow all the way to using Foundation models like Florence to um or and de calling cloud AP like open AI API um and then we've got um various uh blocks for processing results too like detections consensus um State Management across videos um and output blocks that allow you to connect workflows to your uh systems uh then in this uh video we'll be walking through an example showing CSV uh kind of data export from workplace so with that said we're going to kind of use a Yol TR model with a find uh withl workflows and build a kind of end to end application over here I'm going to close this over here I've got this workflow and rlow workflows um this has about five steps and allows us to or facilitates the shipping container OCR use case I'm going to walk through it step by step first we've got this object deture model that was trained in rub flow and on the lv12 architecture next uh we're going to filter all these detections so that we only get the container IDs like as we showed earlier we're able to localize various things the containers the logos the IDS the chassis IDs but for this we're going to identify the ID all the objects we'll filter out the IDS then we're going to run OCR on them so that we can build this endtoend inventory system and so that way we can take photos and have a f list um of the containers and that have been identified within those photos next we're going to use the dynamic crop block which allows us to take the um filter detections and split it up into several different images and so if a if an image contains eight different shipping containers and each one has an ID it's going to split those up into eight separate images and the reason we do that is um our OCR quality can be better if we're localized on very specific regions rather than um kind of passing in one b image um and processing it that way in addition if we were to be running this on a real time video um we can uh use some workflows features to only retrieve new detections which would allow us to given a video only find all the new container IDs um for this uh we're kind of using open ey for OCR with that said if you need something that runs on device we would recommend something like Florence 2 which is supported emble plus dead at deployment environments as well as on device um with workflows and rlow inference and our prompt here is just reading the serial number and only returning that from from a a cropped region and then finally we're going to send our data to a CSV or we're going to format it in that style and so once we have that formatted data we can just save it directly to our device and or P it anywhere we want we are working on several other connectors including a connector legis data to Microsoft SQL database um and several other places too already you can send slack notifications emails SMS messages um and more with the results from previous stages in your workflow so together and these kind of five steps comprise our inventorying system that allows us to detect container IDs and run LCR on them and then save them as H in the CSV format so here I'm going to run on this example image which I'll show this is from our data set test set and it shows three shipping containers with several IDs in there and already our workflow is done this output two here corresponds with the CSV format every um CSV formatted uh record has an Associated Tim stamp which allows us to um map when the results were taken and you can see we've got these container IDs like the top one JB 282 862 identified successfully we got the second one two 26 uh 2 26270 and we also have a duplicate because the containers have horizontal and vertical IDs and this is where H you look at maybe using our custom Python block to do entity reconciliation so the duplication and like making sure all of them have the requisite prefix if you need that um and any of that format thing but here we can see these like cropped regions that been process an OCR has been run on and just so that we can validate okay our model is really doing this work on isolating these characters for OCR and you can see your predictions from roof flow here um and then finally if we uh scroll down yeah we'll see all of those results um so yeah we built this uh application in workflows h this only took me a few minutes with that St we can always extend it too we can add more models we can use data set storage blocks uh we could even H kind of add an active learning component to this where uh we use a rub flow data upload block um and uh selectively sent images back to our data set um in real time and so when this application's running we can be consistently sending images back to our data set and for use and training Future model versions and so I encourage you to play around here see what blocks interest you in are relevant to your use case um and we have everything from these like classical like template matching algorithms all the way to the state-ofthe-art models like L to YOLO V12 segment anything uh to add more and so with that in mind um that's all I've got in terms of demo and you can join us uh kind of next week and future weeks for and webinars on more State of-the-art models more ROFL platform features and everything else at roof.com weinar