OpenCV - Python plays Grand Theft Auto 5 p.2

Key Takeaways

Hello everybody and welcome to yet another Python plays a GTA 5 tutorial. In this tutorial, what we're going to be doing is just kind of building off the last one. The last one we proved that yes, we can actually stream this game to OpenCV/Python at a rate of about 10 frames a second. Um, also just in case that's not you, like for example, if you're on like a probably like a I don't know if you're on any computer that can play GTA 5, I feel like you should, unless your computer's horribly imbalanced, have a processor that can stream that image back. So, um, so I'm playing Grand Theft Auto 5 in an uh it's 800 by 600. Um, so, so that as long as you have a computer that can normally play Grand Theft Auto 5 at full screen on like high settings, uh, you should have probably a processor that can stream Grand Theft Auto 5 at like 10 frames a second, which to be honest, it's actually pretty impressive now that I'm like thinking about it because I'm I'm almost positive that all of Open CV is actually on the CPU. So, that the CPU can play Grand Theft Auto 5 at 10 frames a second is actually kind of cool. Anyway, um yeah, let's um the next thing that I want to cover here is um okay, so we used OpenCV to well actually we didn't even use all we've done is actually used Open CV to we did convert the color but we also um are really just using it to show the window. So So we haven't really seen that. Sure, we could do some manipulations on this and not sacrifice like five frames a second. Okay, so the next thing we need to really test is whether or not we can actually do OpenCV stuff on this data. So, um just in case you're not familiar, uh you don't have to go through this series, um but if you want to reference, um some stuff with OpenCV, you can go to python programming.net, um data analysis, image and video analysis, and bada boom. And then this is just a bunch of stuff that we can do with OpenCV. This is just a bunch of examples, but if you want to kind of get comfortable with doing things and especially like masks and regions of interest, um you might want to check out um check out OpenCV uh the tutorials. Anyway, um so the first thing that we're going to do is just kind of do some typical processing of the the actual screen that we're pulling here. So, um I think what I'll do is eventually we'll get rid of this uh print just because actually the act of printing um especially in idle is especially over a long period of time is going to negatively impact our frames a second. So, um you won't want to always have that there, especially if you're running an idle. If you're running through like a command prompt or terminal or something like that, it won't be as impactful and you you could leave it there for like debugging purposes. Um anyway, uh let's go ahead and do some real OpenCV tasks. So I'm going to define process image. Um and then we'll pass image into it. And then we're just going to say the original image equals image. So the reason why we're doing this is and in fact we probably could just leave it as image, but I like the name original image so it's clear but we are kind of duplicating things. Ah, we could call this original image. I could change change that. We could do original image like that. Then we're not copying that image actually. Okay, we'll do that. So, we'll call it original image. Now, what we're going to say is processed image. So, why are we doing this? So event what happens when you t a typical kind of workflow with OpenCV is you have an original image and then you dumb that image down crazy to like do something like find edges or even more dumb like in our case what we're going to do is we're going to attempt to use OpenCV to find lanes in a road. So, as you um do that, you're going to simplify that image so much so that eventually you just have two lanes, right? But then what we kind of want to be able to do is take those two lanes and put them back on the real original image. So, you always kind of want to save the original image. Um so, anyway, original or process image now will be cv2 uh convert color. We want to convert the color of original image. I'm pretty confident that doesn't happen in place. We're going to find out. But anyway, hopefully it doesn't. Anyway, cv2.color. Um, and then this will be BGR2 [Music] gray. Um, we're converting to to gray. We're converting to cray. Uh, we're converting to gray because we want to have much simpler data. So BGR is um you know it's it's a it's a um an array of values from 0 to 255 right so it's three values per pixel whereas with gray gray is simply a you know each pixel is a single value it's obviously much simpler and we can feed that directly through you know a convolutional neural network or something if it comes to that initially though my my plan is to try to use open CV and um more like rules first before we try to you know throw you can't just throw neural networks at things. That's not how it works. So, so, so we really I'd like some sort of way to to to determine whether or not we are in a lane. And even if we're not 100% like even if we don't have 100% perfection at that, what we can do is every time we have two lanes that we're very confident or two lanes, we can take that data and save that data to later train a neural network. So, um, anyway, that's why we're doing that. The other option too is to just play Grand Theft Auto for like a few hours. Especially like if we just drive the same road over and over um and then take the image data and then the input that we made as we drove that road and train a neural network off that. That's possible to do. And in fact, just recently I saw Nvidia released a thing where you can actually do that. Like Nvidia will log basically what you've done and then you can train a model to play the game like you play it. So, um I'm very curious about that. I wonder how much training data it really takes, but interesting concept. So, anyway, back to what we're doing. Um process image. So, we converted it to gray. Now, what we're going to go ahead and do is just do like some typical canny edge detection. So, we can say we're going to just keep the processed image. Processed image equals cv2.canny. Um and here we pass the processed image. And then you have two thresholds. Um, so we'll say threshold 1 equals 200 and then threshold 2 equals 300. You can tweak these numbers. Um, probably a lot is going to depend on the game. A lot will depend on just so many things. So um, this will be tweakable for sure. Um, now let's go ahead and return the process image. And now what we're going to do is under screen we'll keep we'll keep that happening. Um, and in fact under screen we'll do this. So we're we're going to say now new screen equals process image original image. And then now rather than showing the original screen, this one, we'll just comment this out. Eventually, we are going to show pretty much the original screen with some new things drawn on it. So, I want to keep that there. Um, but for now, we're actually just going to do cv2. Show and we'll call this um I don't know. Well, we we we can call this window. And let's just call this window two before we forget. um window and then cv2 dot um actually we don't need to convert anything. Let's just do new screen. Cool. All right. Let's save that and let's run that and see how we're doing up to this point. Move this out up. New screen. Process image. Original image. Nope. We need to process screen. Derp again. Beautiful. So here we have the edge detection. So this is a little unfortunate. I wish I could make this I'm not sure why this is so big. Maybe we'll have to resize this to eight I guess 86 because what's happening is my monitor is a 4K monitor. So this this is real 800 by 600. This or I'm sorry this is a scaled 800 by 600 to like fit the monitor because the monitor does autocaling. Oh dear, what have I done? come back. Ah, we've been trolled by Windows shake. [Laughter] Uh, let me pull up the script again. There we go. And again, actually, we're still seeing 10 frames a second, which is actually really exciting for me. I'm very glad that we're we're still hitting that. Anyway, we have edges. Beautiful. Beautiful. Can you imagine playing GTA with edge detection? That would be pretty brutal. But yeah, we still are operating a really good frames a second. So very cool. Very cool. Now, the last thing that we need to do is a thing that I kind of put off initially and again I've already I know the solution, right? Um, and that is sending input to the