RF-DETR Segmentation. Benchmarks, Inference, Training | Live Coding + Q&A (Jan 29th)

Key Takeaways

Where is the video? That's a very good question. Hello. Do you guys hear me now? I hope so. Okay. Okay, we are back up and running. That's awesome. Sorry for that. Uh yeah, as usual, I was running late for that stream, so it is what it is. But I'm prepared. I'm prepared very thanks to thanks to the magic of AI. Uh so we should be we should be fine. I have one uh display less this time. So I will be juggling the juggling the uh displays uh in the background. Uh yeah, I needed extra webcam as I want to show you the uh RDTR segmentation in real time. Cool. So, uh let's get started. So, I have several things that I want to uh show you and tell you right now. Uh the first one is that last week uh we released a new version of our DTR uh package. Um and in that new version uh version 1.4 we've released uh segmentation models. So right now we have um segmentation supported uh out of the box uh in the package. And uh if you compare where is the segmentation chart? So it's a it's a state-of-the-art segmentation. We uh uh got over YOLO 26 released I believe two weeks ago. Um we have uh one two three four five six sizes. Uh all of them released under Apache Apache license. Uh so uh you can take it, you can use it in your enterprise projects. Uh you can pretty much do whatever you want with them. Um like I said, I will be juggling the windows a little bit because I don't have an extra display. Um so so there's that. And today I'll show you how to run that model. We will run that model on the webcam. we will uh train that model uh in the collab and we will um I will also show you how to combo that model and uh run that uh in conjunction with uh trackers library. So um trackers is another uh project that we released this time um uh this week and trackers can work uh with um with RFDTR segmentation and detection and I'll I'll show you how to do that today. So we have uh a lot to unpack. Uh so I think uh let's get started. Ah, by the way, one important news. Uh, RDTR paper uh got um uh got accepted into >> a lot to unpack. Uh so I think uh let's start. By the way, >> um sorry, we have I have some technical uh difficulties today. Okay, sorry for that. So that's inconvenient guys. Sorry for that. Um so I just noticed that uh yeah let's make sure that yeah I hope that you will see uh the paper right now. Uh something was not working uh unfortunately but heck is happening. Okay, cool. Jesus. Uh I'm sorry for that. I hope that uh you guys are um are okay uh and still see the screen. Uh okay. So, uh, RDTR got accepted into, um, into a ICR conference. Uh, that's a pretty big conference, uh, um, in computer vision, but even more broadly. Uh, so yeah, I hope that we will, u will, uh, get recognized over there. [snorts] Um, but yeah, uh, today um, like I said, we will train the model. I will show you how to train the model. We trained a uh model last uh time on the stream that was uh YOLO 26 segmentation model and today we will train RDTR segmentation. So uh that should be cool. Uh we will train that model in two ways. So uh we will uh use uh a Google collap uh where of course you can train the model for free um and we will train that in the platform and it's pretty much the same onofflow if you have free account u you get a certain amount of credits and you can use those credits to train them all as well. So uh let's get started. Uh you can uh find the link to this Google collab uh in our um uh GitHub. So we have this um uh really cool repo called Roboflow notebooks. We store pretty much all the notebooks for all the tutorials, all the uh streams in that um in that repo. And over here you have two colabs that we'll use today. We'll use how to track objects with RFDTR and by track and we'll use fine-tune uh fine-tune RFDTR segmentation custom data set. So uh let's get started to as usual uh when we train uh models in Google Collab the first thing I always recommend is to run Nvidia Smi just to make sure that we have access um to uh GPU acceleration in the Google Collab because that's you know not obvious. Um so uh what am I running over here? Uh I'm running Tesla T4. Yeah, let's you know that's a pretty modest GPU. So, let's let's try to make it run on uh Tesla T4. Um, next up, I'm I'm just setting up uh some Oh, I'm Jesus. Sorry, guys. I am a little bit a little bit crazy today when it comes to running late and being prepared. Um, so yeah, I just accidentally opened the the wrong col. doesn't really matter. Uh okay. So like I said, we will run nvidsmi and then um we will install our dependencies. So as I said before um rfdtr uh support segmentation starting from version 1.4.0. So that's the version that we are going to install. Um before that we only support like a preview. So uh version 1.3.0 Z added preview uh to preview of a segmentation task into this package and RDTR 1.4.0 added like a full-fledged segmentation uh to this package. So um we need to install that. Um that may or may not take a little bit of time. Uh there is one caveat and this is why it takes a little bit longer. So make sure that you will um uh set up the environment correctly. Um it only works with PyTorch up to uh version 2.8.0. So in our case and I assume that's uh that's why it takes so long. Um the environment in Google Collab is probably configured with like a slightly higher version uh of PyTorch. So, it takes a little bit more time because um Peep needs to uninstall uh version 2.9.0 or whatever version they have and install 2.8.0. So, that will you know take probably like a minute or two I hope um not not more than that uh to set it up. And I also uh because I was setting up the local environment for the webcam demo uh noticed that because we depend on transformers um uh we need to pin the version of transformers so that you wouldn't install version uh find 5.0.0 because transformers like redo a lot of things uh in the latest version. It's a major release. So, um it's uh you know it's it's u a little bit more chaotic. We need to uh make sure that we import proper things and probably they move things around. So there's that. Uh yeah, it's still installs. I I was trying to uh give you as much information as possible. Uh so maybe in the meantime I'll show you the data set that we are using. So this is the data set that uh quite recently I used to um train YOLO uh V uh sorry we there is no V anymore in the name of YOLO YOLO 26 um so that those this is like um images of cars with like a specific parts of the car uh being segmented. So that's the data set that uh we are going to use today. And as a matter of fact when we wait for uh the training to start in the column I mean the training uh oh the installation completed. Okay ignore that. So just to make sure let's uh peep uh list um all the dependencies and grab torch. So, you know, for those of you who are not like super familiar with Linux commands, uh I'm taking the output of uh peep list and I'm I'm specifically looking for um lines where the word torch happens. So, we have uh version uh 2.8.0. That's great. Uh and let's to take a look at the transformers. Uh and transformers are at version below five. And this is this is great. Okay. So now uh I'm uh pulling few example images that uh I usually use uh in my demos. So uh this is of course me and my dog. Um and we'll use that to run uh first of all the inference and the collab and later on we'll run the same inference on the webcam. Um so to do that I of course need to uh uh import few things and I'm importing from RFDTR package. Um so in the readme of uh RFDTR project um when you scroll a little bit lower to tasks we have detection we have segmentation. So here is the example on how you run the inference with the model. But here uh is a table of all the sizes and uh to run a specific size you pretty much need to import one of those classes from uh our FDR package. So um as we uh run segmentation so we in uh we pretty much um imported our FDR seg medium but if we would like to change uh that uh model to a different size here's what how we can do that. Yeah. So uh we imported that and we run the optimize for inference function. This will make sure that the model runs faster kind of like per frame. Um, oh I I didn't know that you are asking for the arxif. So if you did, sorry for that. Um, here is that. Um, oh yeah. So if we send links then yeah I'll sell the link to this and I also send link to the Google collab that I'm using to show you the fine tuning. So here it is as well. So I'm sending uh the link to the collab that that I'm using. Cool. So imported the model, we optimize its parameters for inference. Now it's time to run it. Uh but before we will we'll run it, I'm I'm setting up few utilities. This util is pretty much to uh visualize the result of the model because the model will return detections, you know, mask boxes, confidences, uh class names, stuff like this. But you still need to visualize that on the image. And for that I'm using supervision. Uh so those of you who used RobF open source stack, you pretty much probably know supervision. Uh and now when I'm run it, I'm running it. Uh you will hopefully see a result. Here it is. So uh it's a standard uh pre-trained model on Koko data set. So all the classes that you know from from the Koko data sets um are here. Cool. Um and now uh it's time for fine-tuning. So we'll pull um the data set uh in this case uh in the collab it was uh using cracks data set but like I said we will uh we'll try to fine-tune on our own data set uh over here. So to do that uh we probably need to have like a full-fledged link to the data set [sighs] uh plus a version of the data set. Okay. So let's create a version. Um which size will we train? Probably nano I think. So let's go to versions of data set. uh versions. Yeah. And we'll create new version. And now as I will train a segmentation model, our DTR segmentation model and we'll probably go for nano. I will make sure that the resolution uh of the image match um the resolution in the readme. Uh so 312x 312. You can probably go uh with like 640x 640. But what will end up happening is that model will probably train uh with more memory. I'm not sure. So don't quote me on that. But definitely doing stretch plus the resolution uh should be fine. So we are going straight for this um and continue and create. So we'll use this version in our finetuning. So let's just give it a few seconds to uh be ready and we will we will use okay. So we have it. So I'm copying the URL. I'm pasting it here. And I'm not sure but I will change that to universe because the app is uh kind of like my own link that I can only see and the universe is the link that everybody else can see. So now we will pull the data set and importantly RFDTR opensource version uh right now support KCO and YOLO data sets. So be before we only supported KCO but with version 1.4.0 you can also use YOLO data sets. So if you are using like uh ultralytics YOLO uh setup now you can just copy your data set and paste it here. Um uh people were asking can I use RFDTR small or medium to train with my custom data set? Is it open to train? Yes, it's completely open source. It's Apache tool uh for segmentation uh entirely and pretty much for any object detection sizes. Uh so if um if you run segmentation, it's uh all of those weights are under Apache. uh if you run a detection uh all of those weights are under Apache and those are under like a custom license. Uh we should probably add that information to the table. I will do that after the stream so that's more clear. Uh but yeah, it's totally open source. You can you can use that. You can use that in their enterprise project. You can fine-tune it for free in the collab. You can finetune it in the in [snorts] the product and it's open source. Uh yeah. Yeah. So, XL and 2XL uh segmentation are under Apache. Uh XL and 2XL detection under that that that other license. Uh does it work with small object detection uh and segmentation like 20 uh by 20 pixels? Uh [snorts] I yeah I mean we tested that we benchmarked that it works but if you have smaller objects then I recommend you will use higher sizes larger sizes because what what happens is that if you have those uh smaller sizes uh we run by default this is detection. Yeah. Um this is segmentation. Yeah. So by default we run under that resolution. So if you will have small objects 20x 20 and you will I don't know resize the whole image into this resolution that 20x 20 object will suddenly become like 2x two or 3x3 pixels and that will be too small but if you will use those larger sizes the speed will be a little bit slower. It's still real time as you see it's like for Excel the latency is 13 milliseconds. So if you will divide 1,00 by 13.5, it still runs at 75 frames, 74 frames per second. But because your resized image will be pretty much having four times more pixels uh than the nano one, it will be a lot easier to detect those smaller ones. Yeah. So that's that's that's important. Um cool. Let me just start that training. So we downloaded the data set and now below we'll start training. Uh to save time I will train uh only few epochs. I'm not even sure if we will train five. I will say we'll train three because what I would like to have is I would like us to be able to Okay, let's try five. I hope we'll we'll make it uh on time because I would like to still show you the result during the stream and and not wait long. But I will still play with the webcam. So So maybe we should be fine. Okay. So to train it's very easy once again uh you import the model. This time you know before we were using medium to to show you the inference because it doesn't really matter but uh this time I'm using nano for the training and it's only to make sure that uh it will train faster. Um and uh what GPU do we have? Uh what GPU do we have? Let me just check that. Okay, this time we have L4. I'm tempted to try to run with like a larger batch size. Let's try eight batch size eight. So, because uh RDTR depend demands like a little bit more um GPU power during the training. Um yeah, I I am a little bit anxious to run with eight batch size, but we'll see. Maybe maybe it will be fine. Um so yeah um we'll run eight epochs with batch size eight. Let's let's take a look at our uh GPU consumption chart. Uh the training starts. I'm scared. I hope everything will run correctly. I don't we don't see a big spike over here yet. Uh but I bet we would use more than two. 2 GB of uh VRAMm. It's weird that we only use this. Uh but this is okay. Okay. That that sounds a lot more reasonable to be honest with you. We could straight up run 16 uh 16 epochs, but we should be able to run the the full training like like I said with only five epochs and we are almost at the end of epoch one, so we should be fine. Cool. Okay, so the training in Collab is running and obviously we cannot make it faster than whatever is happening here. So, uh let it be let it run in the meantime. Uh the loss is going down so that's good. And uh let me just take a look at the chat. latency is still very good. Do you recommend using larger model instead of the sahi library for smaller object? So the question is should you use like a larger model or should you use you know something called sahi we call it um inference slicer. So it's pretty much u uh like dividing the image into smaller um smaller sizes. So I think uh that it all depends on the resolution of the image. If the image that you are um running on is like a very very large image like I don't know 10,000 by 10,000 pixels or 50,000 by 50,000 then you need to use sahi. If that's a like a 4K image, a 1080p image, uh I think that you you should be good with like a larger model. Yeah. So that's how I see that. People uh are saying that I should chill out a little bit. Uh, I will try. I will try. Uh, like I said, I was a little bit underprepared for that stream and that's why maybe I'm a little bit uh a little bit stressed out. Uh, cool. So, while the model is training over here, uh, I hope that we'll get the results soon. Uh, let me show you how to run the same training in the platform. So, how do we do that? Uh once again we create a version because that version that we are running right now will be tied to uh the version um that we train in the collab. So I'm I'm kind of like setting up another version. This one will be tied to um the training that will run in the platform and after a few few seconds we should be good to go. Yeah I people like my answer so yeah uh yeah exactly. So if the if the image is very large than sahi, if the image is 4K 1080p, you can go for like a higher um higher resolution of the of the actual uh model, which in our case means like a larger size and we should be good to go. Cool. So I'm I'm I'm in the in the version of the data set. I'm create clicking train model and here I can uh pick RFDR. You have different sizes. People were asking is it open source? You you see even in the platform if you train it's still Apache too. So you can train it, you can use it. And now I'm picking nano and I uh start training. So that training obviously will take more time because here we'll probably run definitely more than 100 epochs on this data set. In the collab, uh, we are running only five ebooks. Yeah. So, it estimates the training time for like around 4 hours 20 minutes. Um, yeah, I'm not I'm not sure about that, but it will definitely take a little bit of time. So, um, let let's keep it running. I will show you the progress at the end of the stream. I guess we'll still be still be training. Um, okay. Okay. So, how is our progress in the uh collab? We are at epoch 3. Perfect. Cool. So, now let's switch to um local setup and let me show you how to run it on like a webcam. So, before we do that, let me just show you the documentation. So, we updated we heavily updated our docs. Um, and I can show you how to run segmentation. Um, and here are examples on how to run segmentation on your webcam using like OpenCV, RFDTR package and uh supervision. So I use that as an input for uh my uh local setup and this is what we have. So uh I have like three free scripts. Let me know by the way if the font in my in my uh Oh, I made it larger. I think I think this this should be fine. So, what we have we have three uh scripts over here. We have a uh webcam uh viewer uh script and I will run it for you right now. So this script will only um so like I said I had like problems with trackers uh not trackers but transformers so that's why I needed to adjust few things but um before the uh before the stream I picked my webcam and I think this will be the webcam that I will be using. So, let me just show you. This is like a plain uh plain um a plain uh view from the webcam. Um and we'll build on top of that script right now to run rdtr. So, at the end, we should be able to run something like this. And you know this is like a much more sophisticated stream uh script sorry because it has more uh information over here. So I don't know here's like a cup here is like an apple here is like a banana. So uh all of that is happening real time uh through through my webcam. And I will show you how to convert that simple script that is only running the uh that is only running the kind of like the stream from the webcam into this um more or less because this is like a much more sophisticated script where you can choose the size and choose the confidence. We'll skip all that because I don't want to bore you with like uh argument parsing, but I'll show you how to plug it into your like a webcam stream and how to build on top of that. So that should be that should be fine. Uh okay. So let's let's do exactly that. So uh let me go back to cursor. And this is my like a webcam viewer stream. Very simple script. Uh what do we have here? We have like uh argument parsing allowing me to pick the uh index of the camera. So um it works smoothly guys. It works. I have like OBS for streaming. I have um you know uh constantly capture of my uh of my um desktop. So it's a it's a webcam plus a lot of other things that are happening on that laptop and it still works smoothly. Uh can we use that for commercial purpose? Yes. Yes, you can use it for commercial purpose. It's totally Apache 2 means that you can do whatever you want with that. You can take the code, you can take the weights and and build build a startup on top of it. Uh impressive FPS to be honest. Um this is actually a lot slower than you would get on Nvidia GPU I'm running on my on my laptop. So this is what it is. Okay. So like I said, um let's take a look at the script. So that's a baseline script, the one that doesn't uh the one that only shows us the the webcam. Yes. So So there's that. And now now let's build on top of that. So uh let's look at the documentation first. U where's the cursor? Cursor is here. Here is the docs. Uh in the docs we have an example showing us how to uh run that script. Um uh how to run that model on the webcam stream. So first of all we need to import those two things. Uh so uh let's do that in cursor at the top of the file. Let's add those two things. But here just to save save time uh and have like a uh faster inference let's lose let's use the nano version. So I'm using the nano uh and now when we use it we need to initiate that model. Yes. So the next step after import is initialization of the model. So by the way maybe let's let's make it a little bit bigger for you guys so you can easily uh read it easier. Uh would like to attach this to drone. Uh curious to see how it copies with speed. Um so can you tell me what is the hardware that your drone is running? Do you run uh like a a board like a Raspberry Pi or something like this or do you run a Jetson for example? So now we take that we will initiate that model. So I'm going back to cursor and somewhere over here maybe below the parsing we'll initiate we'll initiate our model. So so there's that and now uh let's pick uh device MPS. Yeah because uh I'm running on Mac so I don't have like CUDA acceleration but I have MPS acceleration and RFDTR supports not only CUDA but it supports uh MPS. So let's use that and I will add this line optimize for inference. I showed you that line uh in the collab that pretty much make sure that uh the model will run faster during the inference. Uh people are asking about the XL and 2XL sizes. XL and 2XL for detection are under different license but for segmentation are under Apache 2. So you can do whatever you want with that. You can do whatever you want. Um okay I'll I'll go uh further. So right now what else do we need? So we are running segmentation model. So uh let's take a look at the annotation. Annotators that we can we can use. So annotators are those things that that we did in the collab. Yes. So that's stuff that we had somewhere over here in the collab when I was running um this uh this demo over here. So let's copy that. Why not? So let's take this color. Let's take the mask annotator. Let's take the polygon annotator label. Um, and yeah, let's copy this as well. But let's start with this. Let's start with this. Okay. So, let's copy that. So, this will uh handle like drawing everything on the screen. Yes. So, if we will uh and I need to probably import supervision. So import supervision SSV and now [snorts] um that part will need to be slightly in a different place but those annotators should be good. So what do we have? Um yeah have this. So what do we have? We have mask, we have polygons and we have uh label. Uh so we will kind of like draw the whole area then we will draw like a outline white outline around the the object and then we'll draw the label for that object. Yeah. So there's that. Um now we are opening the capture that will take the stream from the camera. Um this and this and that. And now here is the place where we have the frame. Yeah. So we are looping over frames from the webcam and our inference should happen pretty much here. Yeah. So important thing is and because I already have other scripts that are doing that in my environment a cursor is uh showing that but um the inference in rdtr happens on uh bgr images. So if you run po for example like we did in collab it's not a problem you just load the image and pass it to predict because bgr is the default order of colors in um in pillow but if you run open CV then you need to switch the order of the colors uh and there is a function in open cv that does that uh so it it takes the frame and it's give you bgr frame so originally uh sorry uh RGB frame. Yeah. So, so uh there's that and the threshold. Let's set it to something smaller. Uh and this will give us detection. So, this is this is like a predict method that uh is taking the frame and it's returning detections. Uh and the last step for us is to draw everything that we need to draw. Um, so let's also make sure that we will draw the labels. So let's have something like this in here. So we have um cocoa classes get confidence. So we are parsing the labels and the last thing we need to probably pass labels in here. Okay. So ah and we of course need to display display here. So that that should be it. That should allow us to uh to convert we convert the channels like I told you to run the inference. Then as we have the inference, we take the class ID, we take the confidence to form our labels and then we draw everything. So we draw the mask, we draw the polygon, we draw the labels and then we display them. Cool. Okay. I mean I I hope it will run. So let's try. So the model is loading. It's converting and I already can tell you it it's running correctly. Yeah. So the visualization is slightly different because we use different uh we use different annotators than the one that we would use by default. But this is what it is. And this is the nano size. Yeah. So we can now maybe try to run medium just so we see how fast this Let's me let me just maybe try I I actually didn't try like medium size in callup. So uh not in color but locally. So So let's try the medium. So I'm I'm just I'm just switching nano for medium. Uh run the same script. It will uh probably take a little bit of time because we need to pull uh we need to pull the weights. And maybe if we'll have like even a a little bit time, I will show you my favorite annotator and supervision. But um let's see if if that works and how fast is that on uh is there a documentation for running the model on Jetson device? In fact, there is and I will show you how. Okay. So here is medium. Yeah. It's still pretty pretty fast on on local device. Here is the apple. Here's the banana. Yeah. And I don't know. Here is Here's the cup. Yeah. Cool. Okay. So, I would show you my phone, but I'm recording with my phone because I don't have the webcams. So, [laughter] so there's that. Cool. Uh, so so there's that. Maybe before we wrap up the local, there is one cool annotator that I want to show you. Um, yeah. So, we have a lot of annotators, but my favorite one is Halo. So, let's try Halo uh as the last thing that we'll do locally. And we'll maybe convert to N. Oh, maybe let's keep it like this. And instead of mask and labels and all that, I say we use halo, we pass the colors. And here inside instead of doing this we'll just do that halo annotator the screen for the for the last time. Uh it's not so cool. Uh, I mean maybe you will understand why I like it, but we'll see. It's like a glow like not sure if you watched like Dragon Ball or something like this, but like when they converted to like super super scion mode, they they had this like glow around it. I think it's pretty cool. Uh, but it looks better not on webcam apparently. Um, cool. So, there's that. So, there's that. Uh, let me just take a look if there are any questions. Um and if not then then we'll go back to the to the collab. So um yeah medium. That was medium on the laptop. Yeah. Confidence high. Rebounding boxes. Exactly. You're a legend. Thank you very much. Uh can it work on on more tiny CPUs uh or GPUs? Yeah, I mean Jetson definitely. So that's a more tiny GPU. Uh, exactly. Pure super scan mode. That what I was looking for. Okay, cool. So it looks like we don't have like a lot of questions other than that. Is there a documentation on how to run it on uh on the Jetson and then we'll go back to collab. So let me just go in here. So to run it on a Jetson, you run it through the inference package. So if you go here in the docs of RFDTR we show you that you can run it through two things. You can run it through RFDTR package and then you use those classes or you run it through inference package and then you run uh with those aliases and Roblow in Rooflow inference Jetson. So um pretty much how you do it is you go through the installation process that you see here. It will guide you how on on how to install uh the inference package on your machine depending on like different jetpack versions that you have all of those things and then you pretty much run it uh as any other model in uh in here. So if we have uh let's say rdtr. Yeah. So it's so once you install according to those uh in install guides that I showed you here is the here's the information on how you run it. So in here it's a little bit different because you need to in in import from inference get model and then you pass that alias and that alias that you pass is the same alias that you have in the readme in here. So that's how you that's how you pretty much do it. Okay, let's go uh back into collab because I hope that our training is already completed and it looks like it is. So let's print the plots. So we can see that the model was training and we got in just five epochs. So in the meantime I was showing you how to run uh can we run it on tensor RT. If you run it in inference, it will run as a tensor RT model. So it'll get converted. Uh documentation is running [laughter] on video spark. No, no, it's not. Uh Tobius just turned in. I mean, you missed a lot of things. I was uh I was Son Goku from uh from Dragon Ball Z uh in the meantime, and we already trained the model in Col. So, so the training is completed and the model um probably could still train for longer and get better results. But after just five epochs of training on that data set, we got to map around 80 over 80% uh and map mat map 50 95 to around 55. So that's pretty good result if you ask me. Uh if you just train for five epochs. So now uh before I will show you the results, let's make sure that we upload the model um to the uh to the Roboflow platform so that later on we could use that model. Um so yeah, let me just versions. Yes. So we just need to make sure to copy this. uh our data set was version five. So let's just do this and this sorry version seven. So I'm uploading my model to robflow and after you know few minutes it should be uh running uh over there in the platform as well and now I will show you the results after just uh five um five epochs. Uh I mean you can still I mean it will be uh accessible as a video after the stream. So no worries you can rewatch that later on. So it's uploaded Roblow tells us that our model will be available here. H and it will be but it will take a little bit of time. You can see that it's already syncing. So we just need to process it a little bit. By the way we are here. You can see the training. So after five epochs in the collab, we were at eight over 80 map which is here and after nine epochs in the platform we are already at 85 map and map 5095 is still growing quite heavily. So we are at like 57. So the model is converging very fast. That's what I'm saying. That's what I'm saying. It's training. It's training quite fast. Okay. Now let's um clean up everything after the training so that we can have empty GPU and we'll load our fine tuned model um into the collap. So uh let's do it. So I was using nano. So I'm once again uh using nano over here. But I'm passing the path to my fine-tuned weights. So the model is being loaded. It is now I'm using supervision to load my test set from the data set that I was training on. And now we can run the inference on like subset of those images. So we picked like 16 images and and this is what we get. Yeah. So um so those are the images. Now maybe we'll take a look at like one of those or two of those images. Uh annotated images. Let's say didn't copy annotated images. Let's say zero. So this is what we got. The quality of that image is of course quite low, but that's because the the kind of like the size of the input image is very low because if you remember at the beginning of the stream, we resized every image to like 312 by 312. So, it was a pretty small image, but we can see that the hood is there. Um, we have the wheels, we have the mirrors, uh, grill. So generally speaking, uh it is already pretty good at the segmentation of those parts. If for those of you who were here for the YOLO 26 stream, you probably remember that the YOLO segmentation model was actually like converging quite slow. So there's no way uh it would look like this after five minutes. Um okay. And now we can also see that our model is already in the robaflow. So now I will show you how to uh run that inference with that model. So so what we did is we trained the model using rdtr package uh in the collab for free and then we uploaded that model to robflow. I also showed you how to run the inference with like a rfdtr package the the open source package that we have. But now for those of you who for example would like to uh run that inference on the drone with like a Jetson nano strapped to that drone. Uh I will show you how you can then take the model that you trained here and deploy on your drone. Um already approach uh approaching the plateau. That's probably expected for simple data set. Yeah. So it's already approaching the plateau, but if you would hold it for like 100 epochs, you will probably get like three, four, five map more. Yeah. So that's the difference. You get to like 90 90 95% of the effect with uh just five epochs and then another 95 epochs to get those like extra three, four, five map points. So, so that's what you usually go through when you fine-tune like small data sets. Um, so the inference is installing here and I and we'll be able to run it in just a second. In the meantime, because I already promised you, uh, I will set up the collab for the tracking. So I'm just I'm just making sure that uh the dependencies will install and our inference is installed already. So like I'll show you in the inference uh documentation to to load the model that you have um on Roboflow or one of the pre-trains model that we have in Roblow you need to use the get model function. So that's what we are doing over here. So we have like from inference import get model and then we need to have model id and our model id is here. So this is the name of the project slash version of the model. So in our case that's it. And because that's not an open-source model it's my private model the one that I train. So I need to have API key to pull it because otherwise everybody would be able to pull um your model import cache tools. This is crazy. That's not good. That's not good guys. I think we'll not be able to run the inference this time. I will I will not try to debug uh the inference package uh during the stream guys. Sorry for that. So um yeah, you can trust me on that. It it works. We'll fix that. Uh there are two ways to run that but not today. Not today. The the inference kind of like died on us. But uh we have the RFDR package that inference work. And fortunately um the tracking demo that I wanted to show you is also using the RFD tier package. So we should be good. Uh we just need to wait for that installation to uh finish properly. Like I told you, it's it's because um it's because we need to uninstall the PyTorch to install uh the other version of the pipe. People are saying install installs install this dependency. Okay. I mean let's let's try let's try let's just have like a one step to fix that. Okay. ppin style cache tools require is already satisfied. I think it's something with the import cannot import name this one from this one. It's probably it's probably somewhere else. Honda conds to be case. So, you know, collapse race again. That's that's exactly what's happening. They have some version. We have a different version and that's it. But this should work. I promise that this should work. I just love the fact that you guys are not pissed at me that that final part of the demo did not work. It's not my fault. Uh I'm sorry for [laughter] Okay, cool. So now we installed everything that we need. So we installed the inference with the GPU support. We installed the trackers package. So that's uh the package that we develop at Roboflow. Um and we recently released by track. So that's the tracker that we'll use today. Um and I pulled some example videos um of like motorbike dirt motorbike and we'll we'll track them. So first what we'll do is we'll um Ooh, it's also using the inference. We'll we'll we'll we'll redo that. We'll redo that. That's not a problem. I will plug the tri we'll use the RFDTR version uh the PyTorch one, the open source one. If if that's [snorts] uh if that will bail out on us as well. We'll see. I mean, I'm glad you guys are having great time. I am sweating right now. First of all, because I'm closed uh in this room. Uh and second of all, because um nothing is working as I wanted it to work. But okay. Uh cool. So, let's see. Let's see how it works. So, this is the Okay. So we loaded the model. Now let's try run the model. So it looks like it only fails to run when you uh use not the open source model but the uh model that is in your project. So we'll see the video is being that is good. That works. Great. So now take a look. We were running tracking on that video and we have uh here is the outcome. So this is the detection model. So this is by track plus RFDTR uh not the RFDTR segmentation but by track plus RFDTR. So there's there's that. Now let's try to run the segmentation model. So as I told you before, how do we run segmentation model? Well, we need to load the segmentation model. So we go to the documentation page and we'll see. Okay, I would like to run the medium size. So we go back to collab, we switch this to medium. So the only difference is it was RDTR medium, now it's RFDTR seg. And that should load the segmentation model for us in collab. Just a second. Just a second. Let me just take a look in the um in the chat. Okay, maybe I will read the chat a little bit later. So the only difference that I will make here is I want to use the mask annotator as well. So we go back I believe here. Where is our annotation new? Okay. So we will take this. We will take mask annotator and polygon annotator. We will copy that somewhere in here. And now what we'll do is we'll take this line. Copy that. I need to run this. So we'll do mask annotator polygon annotator. So I'm doing you know. So we switch the model and we'll run the inference. But we didn't have any annotator that would display the masks. Now we have. So now we can run it. Yeah. So once again it runs. So in collab that segmentation model is running quite slow I would say but um when you run it as a proper tensor it RT it should be a lot faster. Okay, let's take a look at the chat. In the meantime, also were te tested by track with RDR segmentation work great. Uh GPU heating up the room. Uh yeah, I mean it's it's the lamp that's heating up the room, guys. I have like a huge lamp straight into my face that is having its own ventilator even because it's heating up. Don't worry about it not working easy uh with uh I'm sure it understands how it goes. Um tracking I have not seen before. This is cool. This is very cool guys. This is very cool. Okay. Uh I will go back to chat in a second. But you can see already we have those we have those outlines. We have those masks. So now I will just run those two commands and we should be able to display the result in the callup. So yeah, this is this is pretty much it. We have segmentation plus tracking with RFDTR SEC plus trackers library. So there's that. There's that. That's pretty much pretty much all I wanted to show you um today. If you have any questions, now is the good time to ask those questions, guys. I will just take a look at the chat in the meantime. Um why I am not able to use and train RDTR small with my custom data set through Python package. That's a very broad question. Uh you should be able if you have the Koko data set or the YOLO data set, you should be able to train it. Um, but if you still can't, there's probably something up with um the processing of the data set. So feel free to open the issue um issue in our RFDT repo. So if you have any issues with training with that package on your custom data set, here's the link. Go into issue section and please open the open the issue and and we should be fine. Um what else? Uh it could be combined with the timing sensor to capture and watch and do races. Absolutely. We have we have like maybe I should have did you guys saw our like how to count object crossing the line tutorial. I recorded that like three years ago. Um maybe we should redo that tutorial with RFDTR and by track right now. Uh would you use by track in problem like the basketball analysis that you previously made? Um no because by track is not good enough at handling occlusion. That's why we use sam tool but sam tool is like probably like a 20 times slower than by track uh because it takes like a single second to process a single frame. It's probably way slower than than 20 times probably 100 times slower than by track. So uh by drag is great for a lot of things. Um SAM tool is great for a lot of things but you need to pick the tool that you are using and if you want to run um good accuracy then it's probably some tool. If you would like great accuracy it's probably some tool. If you would like to have a good accuracy and the speed then it's probably brack um I heard that the team is working on magbite. Yes, we are but we are first we'll do OC sort but magbite is coming so it will be very strong tracker compared to other trackers so uh stay tuned for this we will have another release of trackers in February and probably another in March so we are working very heavily over there uh link for tracking collab incoming guys wait a second um this is the link for tracking collab Um and by default is object detection. So you need to do those changes that we made today to make it working with segmentation. So load a different model and uh and you know change your annotators from supervision to run the uh annotators for segmentation. Uh what do you think about uh cascade detection with uh multi- RDTR models? A small model for initial Roy crop validate detection crop with large model. Yeah. I mean we did a lot of like you know things like this in previous projects in basketball AI for example what we did we we did kind of like c cascading flow because we had like a player detection and then we cropped the player and then we used that crop to um analyze embeddings to to figure out which team that player is playing for or we had like detecting numbers on the shirt so that then we can crop out those numbers and then [snorts] we can use another model to uh classify that number. What number is that? So uh those flows are great but of course any time that you do multimodel flow then everything is getting slower. Yeah, that's that's pretty much what it is. Um can we pass GPU transfer to Python API? No. Would you like to? Because if you would like to then we can probably make it happen. I think uh feel free to make more streams or detailed videos. Maybe I will do maybe maybe we'll do another stream where we'll do like RFDTR plus trackers plus like advanced things like counting objects crossing the line or counting objects in the zone. We made those things very far away in time. So maybe it would be pretty cool to to do it once again. uh data set in scooco format. I can train RDTR sec preview but not the nano small or medium says cannot import. Ah okay so your problem is that you I told you guys in the stream you need to have RFDTR 1.4.0 zero installed. And if you have access to SE preview but you don't have access to latest models, it's because your uh Python package is outdated. So you need to update to the latest version of RFDTR and you should be able to train no problem. uh that would be amazing uh to pass tensor on GPU were part of the high path pipeline uh with data saying okay I mean uh that's a valid point like if you already have that frame on the GPU and you would like to pass the tensor we will probably try to make it happen let me just check okay because maybe we can already do that uh where would it be RF DTR. Um, maybe we already can code. I'm a little bit scared to check that like during the stream right now, but We will see. >> Okay. Yeah. I'm not sure I I if you will um write an question on RFDTR um GitHub repo, I will try to answer that question because and sorry you you guys are probably like thinking, why is he doing that? Because my webcam is in front of my display. So sometimes some things are included and I need to kind of like look like this to take a look what's behind it. So sorry for that. So once again, if if you uh want us to allow passing GPU tensors into the predict function, uh open an issue on GitHub, please, and we'll try to make it happen. Um [snorts] if you use segmentation would a post model no longer be relevant? So pose is a completely different model like you can segmentation can tell you where exactly things is things are but if you for example would like to know like when you are shooting the ball in basketball for example it's very important that your wrist is in a certain way and you wouldn't be able to know that with segmentation but if you would have pose of here in the wrist and here in your fingers and here in your elbow you should be able to exactly know like the angle of your hand for example. So segmentation model will never give you that but pose estimation model will give you that. By the way we might release our DDR uh key points. So you know stay tuned. I'm not saying we will but it's very possible. You mentioned some tool uh high accuracy tracking. Does some free also handle tracking similarly? So sum to two and sum three have very similar tracking capabilities. The only difference is in sum tool you could only track by providing a like prompting with the bounding box. In sum you can prompt with text and it's slightly better when it comes to like accuracy of the tracking but speed-wise though those are very similar. The only difference is like how you prompt the model. Uh the RDTR is uh fascinating and I'm pretty interest I hope that you will like the model. Spread the word. Let people know that there are other models than YOLO that transformers are uh getting better and better and are actually stronger than um CNN's right now and uh yeah do it. Um I guess postprocessing would be adapted. Yeah, I think I think that that part when you would like to pass uh torch tensor into the infer method, it's it's not that hard. We should be able to figure it out. Uh I will sending off. Thank you. RDTR post would be a banger. I think so too. I think so too. I we'll see. I will try to convince our model team to make it happen. Uh I can't wait to build all of those like push-ups demos where I can count how many push-ups I did. I'm terrible at push-ups, but I have wedding coming soon and I need to get slimmer for that wedding, you know, and have a little bit more pump in me. So, I need to do push-ups at home. So, I will just create like a pose estimation app to count those when we will release a pose estimation model for RDTR. Um, can you make a separate stream for RFDTR segmentation with tracking? probably if we will add like counting objects and stuff like this then why not uh how does the segmentation model architecture differ from it's completely different than yellow yolo is CNN is like a convolutional neural network um and is transformer so literally completely different so there's that uh nice life coding thank you very much congrats on the wedding thank you very much I will pass those to my fiance say. So yeah, I won't be around in May. I have honeymoon, guys. So no streams in at the beginning of the May. Um, cool. I don't see uh people are voting for FDTR post. We'll try to make it happen. I don't see any more uh comments in the chat regarding the actual model. So thank you very much for being here. Um, spread the word. Use RFDTR, use trackers. uh let me know if something breaks, if you need new features, um and we'll try to make it happen. But in the meantime, once again, thank you very much for joining me. Apologies for like a very very very chaotic start of the stream. Probably the first 10 minutes are hard to watch, but I hope you had fun and uh I will try to make more RFDTR segmentation tracking uh you know, live coding uh streams in the future. So that's it from me. Uh and yeah, see you next time. Maybe next week. We'll see how it goes. I will let you know on Twitter. So So follow me there. See you. Bye.