Determining other vehicle distances & collision warning (object detection) - Self Driving Cars p.18

Key Takeaways

what is going on subscribers and others welcome to the second video of object detection inside of grand theft auto using the tensor flow object detection AP i-- in this video what we're gonna do is build on the last video in the last video we just got object detection working in grand theft auto and now what we're gonna go ahead and do is try to detect cars in the distance basically and then try to figure out some way to calculate their distance from us like are they far enough away that they're not a problem or are they look really close and then from there we can build on top of that to detect when a vehicle maybe it's too close and either take evasive action or just display like a warning or something like that so that's what we're gonna be doing now from the last video since I was recording and noticing frame drops I went ahead and just kind of changed something in here you can designate how much vram the the session wants to take up here by adding these GPU options and then when you run your actual session in the config parameter there you can add in some configuration information so I'm going to try to set it lower I might even go with like 50% I'm gonna try 70% right now and see if that works so so let's go ahead and get into the code so at least on the object detection API if you know for tensorflow that little mini-series people were asking like how do I get that those the information and how do I like you know continue working on that and at least to me I think I feel like it's fairly obvious right here you have your boxes here you've got your scores your classes and then you can also see here down here you've got the actual return from the detection of the session of boxes scores and classes so if you want to continue playing with this I mean this is this is where it begins and then if you want to play with like how it's actually visualized on screen you can play with this function here so basically what we're going to do is before the in show we're just gonna work right underneath the busy till dot visualize boxes and labels and all that and we're basically gonna start iterating through any one of these would work I'm just gonna iterate through boxes I am sure there's a better way to do all of the things I'm about to do this is just an example of proof of concept so anyways what I'm going to say is for I be in and numerate boxes zero so boxes is like an array of or array it's an array inside of another array I don't know why that was so difficult for me to get out but anyways box is zero begins the iteration through all of the boxes and the boxes contain the coordinates I forget I think it's like x1y1 x2y2 but I honestly forget the order I think that's it anyway we'll get there so find out if we're right or not so anyways for IB and enumerate box is zero now what we want to go ahead and do is check the classes of those boxes so in this case we're trying to watch out for cars so so how do you detect the date like let's say you've got a flat 2d image right how could you detect the distance of an object in that image at least for me I believe the only way you could possibly do that is if you knew the size of that object beforehand so you can detect distances like so for example the average car I don't know three feet four feet wide I really don't know let's go with like four feet wide maybe an SUV might even be five feet or something I don't know but let's just say you know it's somewhere between four and five feet wide okay but so if you wanted to detect a car you know it's four to five feet wide so then if on your little image depending on how wide it actually is on the image you could actually did you could actually calculate the exact distance of that object if you knew the exact size of that object unfortunately we don't know the exact size of every single car in Grand Theft Auto but we don't need to know we just need to kind of have a general idea like okay these objects are about these sizes so for example for a car we're going to do is we're just gonna basically measure the amount of pixels in between you know from X x1 to x2 how wide is that and then from there we can get a relative determination of how far away that object actually is but you that rel it like for a car we're gonna wind up using about like 0.1 okay ten percent of the you know the pixels that are taking up the screen let's say and actually we're gonna we're gonna we're gonna end up changing that functionality but let's just say at 10 percent if the car is taking up 10 percent of the pixels on the screen and we're you know traveling at a good speed now it's a threat or we could say if the car is taking up 30 percent of the pixels you know width wise on the screen we're in trouble but that number would be different for say a wedding a pedestrian I mean if the pedestrians taking up 30% of everything that's on screen you just hit that pedestrian so it just depends on what the object is and how wide it is anyway what we're going to do now is so we're gonna ask if classes 0 hi because just like boxes they're like a raised inside of a raised if class is 0 I is equal to 3 which is a car or classes 0 well let's just do this will detect probably three things here and even this might not work very well but anyway three three six or eight so what car is three six is a bus and eight is a truck and you might be like syntax how do you know that so the way I did that or I found that out is let me just navigate there inside of the object detection models yes I'm sure I think was in training no is it in data I forget which one of ten it's in one of these directories if you go to dude I can't find it now if you looked if you look in the actual ipython notebook you'll find wherever it's listing like again if you go to the tensor flow object detection tutorial remember those configuration files I can't even find the configuration file I really want to find it for you guys so I can at least you show it to you but I'm not finding it I'm gonna deleted it or something or I'm blind well both anyway in the configuration file it links to where all the classes are just like when we wrote our own classes so I just went there and figured out what was the class for car truck and bus but there's 90 possible classifications so there's a lot of different things you could add but I figure kind of that car truck bus they all have to fit in standard American lanes so they're probably about the same issue with cars are gonna be less width obviously than like trucks especially a semi truck or something like that but generally they're the same plus to avoid a truck that's bigger so you probably give it more more space anyway so anyway continuing along next we're gonna ask if score scores zero I is greater than 0.5 so we're doing this again because if you can't see it here but if you went and actually go into the visito and then read this function one of the the parameters although it's set as a default is like the minimum score required to draw it so you might notice that none of the objects ever have a score or less than 50% well that's because it just doesn't draw anything that has less than a 50% but actually if you lower that threshold you would be astonished by how many objects are actually being detected on your screen it's it's a large list so you can change that but anyways but if we just iterate through these you'll get tons of objects that have less than 50% and we don't we just don't really want those you can feel free to tweak this a little more if you like especially any especially the further away you get from real realistic depictions of whatever you're trying to detect you might have to change the score a little bit but anyway now we're going to measure the distance so the approximate underscore distance we I could also call this relative distance because we're not actually going to convert this to any metric or imperial units I'm just going to use a relative distance you can feel free to write a conversion if you'd like something that's fairly realistic I'd love to see it because it gets hard as things kind of travel off into the distance and I thought about doing it and I was like there's really no point because we're really we don't care about how things are like way off in the distance we really just want a lot of granularity as we get real close so how we could do it is basically we're just going to say boxes and this will be the the x and the y so I think it's actually like a Y and then X let's do boxes I know boxes 0 I and then this is 3 minus one if I'm wrong someone yell at me but I'm pretty sure that'll work I mean the why would I mean in theory like the height of the thing would work as well but then trucks would be severely like trucks and and like semi trucks or actual trucks and SUVs they really vary more by height than they do by width but anyways we'll see how that works so that would be a way to measure approximate distance but this this could be so this is this comes out in percentage so what's one more thing I wanted to bring up so so the boxes are not in pixel numbers they're actually again in relative so it's from basically zero to one so it will be like point one or point six if it's perfectly in the middle of the screen it will be 0.55 point five so so so this will return this could return in theory a long float and we don't really want that so what we're gonna go ahead and do is we're gonna we would round it this and then we'll just round to one now the other thing we'd like to do is let's go ahead and encase this in parenthesis and then we'll do a one minus that number and then this way because like this way would have been a lot larger as it got closer and but if we do one minus the difference here it'll be smaller as it gets closer so that's just a little more intuitive if you ask me okay so once we have that what we're going to go ahead and do I mean I think that's right let me just make sure that closes off yeah okay now to give it a little more granularity as we get close another option we have is like to encase it like this one and then to the power of four or something like that that'll give us a little more granularity cool so good enough we can kind of tweak this as we go on we'll just kind of see how this how this pans out for us now what I want to do is we want to calculate basically where is where's like the middle point of this this object so we want to we want to do this for a variety of reasons one I'm gonna use it to as a place to display what's the distance of the car but also we want to know where that car is like where's the middle point of that car so we can determine is this car actually in our sights is it far off to the right or far off to the left or is it something we're about to hit and if we wanted to avoid it if it's more to our right then we could turn left or something like that so we we kind of want to know the midpoint of that vehicle so the way that we're going to do that is I'm going to say mid underscore X is equal to and in fact I'm gonna just take this here it's equal to two the x's but we're actually just gonna add them to each other so it's equal to those yeah yeah yeah divided by 2 and then 4 mid why I'm just gonna copy paste mid Y and this will be to 0 okay so so this will give us yeah that's fine so so this will give us the middle point of everything so now what we're gonna do finally is we're going to add a a CV to dot put text and then where do we want to put that text we're going to put it to image underscore numpy and then what we're gonna do is we don't really need the string formatting but I'm going to do it a px distance and we're rounding that but what you could do is if you did convert it to like some other unit is you can say like meters or something like that otherwise you really could just remember this to string then what we're going to do is where are we gonna put that so again these are not pixel values there are percentages this is a percentage as well so but CV 2 dot X doesn't want a percentage it once exact pixel values and we've decided that the screen that we're gonna show is an 800 by 450 so to do that what we're gonna say is this one's XY z-- in tuple form so I'm gonna say int it's gonna be its by an int want to make sure you do into it won't take a float because you can't have like a half of a pixel so we're gonna say mid x times 800 and then int mid yep mid Y times 450 make sure yeah it wasn't off screen so and then we can pick the font so we'll just do CV to dot all caps font underscore Hershey honors or simplex and then what we're gonna do is for the size will do 0.7 the color will do 255 255 255 and then line width will be a 2 so that will put the size or the approximate relative distance on the car hopefully somewhere in or near the box having to be perfect so it's gonna start right in the middle and then go off to the side but it's just for debugging purposes I don't really care to make it that beautiful and then finally we are almost done the final thing we're gonna say is gift a px distance is less than or equal to well just sort of 0.5 and then we'll kind of see how we're doing so it's nice that are equal to 0.5 that object is close but that object if it's like off to our side like a little bit like if it's like near our like front quarter panel of the car it's gonna be huge but we're missing it we're totally fine so so we wouldn't we wouldn't want to always warn just because that object is very large and it's close we only want to warn if it's in our you know our trajectory so we're gonna ask one more question that's gonna if mid X is greater than 0.3 and mid X is less than 0.7 if that is the case we've got a problem then what we want to do is display a warning or something like that so I'm actually gonna just copy this line here copy paste I just want to get rid of these tabs so what we're gonna do is on image numpy yeah then our message is just gonna be warning we're gonna display it hmm we could display it on the vehicle but it kind of neat I was gonna display it in the corner but part of me wants to display it on the vehicle we could display it at minus 50 maybe this should work right we'll see cool and then our color we're gonna make it this is CV two's working in BGR so 0 0 255 would be all red to fine it's a warning let's make it 3 unless they get one point out it's probably this isn't gonna be pretty I'm just warning you in advance it will be pretty okay so what I'm gonna do now is save this and I'm gonna try to run it I'm gonna go ahead and pause because it takes a while for everything kind of fire up and we might hit an error or something like that so I'm gonna pause while I start this up and then I'll get like a vehicle or something like that and then I'll resume once we were running all right let's see if our are 70% GPU usage works better so we're in a car we're detecting cars at distance that's pretty cool as it was passing on her sided and it wasn't a problem but then it became a problem no warning for that guys we passed let's go ahead and head towards these cars we got a warning he was probably a little too late but it worked my minus 50 actually was a pretty good distance there yeah those warnings are way too late I just tried to brake like as soon as I saw them it didn't work but what we can do is just since that guy is no longer with us we can use this car for an example which is apparently a truck but that's okay so it's right in front of us and we're getting the warning but then this car to our left is in bounds but it wasn't given this a warning now he's gone because he we weren't gonna hit him same thing with this car here so this guy if we continue to turn and maybe face this car whoops hit him my bad my bad homie now that original cars not a problem now it's this car and if we continue to turn a bit both of these will probably could be a warning maybe sort of if I think that there we go now both cars are a warning so pretty cool that we can detect other cars detect kind of some sort of have some sort of collision warning system and all that it also looks like the frame rates are better that works so 70 percent seems to be good enough to recording so just continue doing that so might come my GPU okay pretty cool but as I was saying before that's not all we can do with the object detection API there's obviously a lot of stuff that we can do within the kind of self-driving car format but one of the other things I'd really like to do is at least for the stream I'd like to be able to have it you know you know full circle we're on that self-driving car stream if the agent for whatever reason Charles gets out of his vehicle he has no way to get back into a vehicle he doesn't know how to steal a car so I thought it'd be kind of cool to use something like this where we know where the cars are we know they're about distance could we write some code that accurately goes to the car and steals the car if we're on foot so that's what I'm going to do in the next probably two tutorials so you can stay tuned for that if you've got questions comments concerns or whatever up to the point of this tutorial let me know I'll do my best to help you out otherwise I will see you in the next tutorial possibly it's impossible to pause this while I'm in the game