AI Learns To Play CS:GO By Watching Humans Play!

Key Takeaways

bots are pretty stupid sometimes i'm pretty sure everyone can agree on that especially in the games that are much more complex like cs go or league of legends we even use the term bot to describe someone who is terrible at a game and has the same performance as an easy bot pretty insulting to the computers on the contrary games such as chess or go that can be solved algorithmically we would have more of a hard time trying to figure out if the player behind the screen is a bot or an actual person but why are bots in chess harder to distinguish than a first person shooting game we all know that chess is a complex game but how much less complex than cs go is it and making it so much harder to tell apart a human player in a computer you could argue that the lack of expressions in game makes it harder to distinguish but how do we exactly measure these expressive inputs to break down how to develop an ai agent that plays like humans in csgo competitive mode the paper proposed three levels of complexity that require full mastery to achieve it the first level is the short term performance things such as aiming moving and reacting to enemies are at this complexity level this is also the easiest part to make as most cheating programs like aim assistance are at this level while the higher complexity levels are handled by the human users you'll see why the second level is the medium term performance actions such as map navigation ammunition management and reacting to health level are at this complexity level these things are less concerning compared to the short term performances as these skills are not necessary to play the game however it can provide you with an edge to win the game for example if you have played any fps you would know you have to play safe when you are low on hp rather than rushing around to gun down people this is because we know that we have a smaller chance of winning as it will take the opponent less time to take you down in an upfront showdown so strategies such as hiding in a corner to do a surprise attack will give you more of an edge to win a showdown you could say that this medium term performance is like short-term strategies or even a playstyle meanwhile the long-term strategies will be the third level concepts such as economy management planned strategies adapting to opponents playstyles and cooperating with teammates would be the highest level of complexity is there a cheat that helps you to cooperate with your teammates or plan strategies for you definitely not and if there is silver strategies will still probably be more practical than what it comes up with that being said this has definitely been achieved in open ei5 which is a dota 2 ai that was developed by openai they were able to have their own strategies and play styles which had a win rate of 99.4 with a staggering 7215 wins and 42 losses in total in a semi-restricted environment open ai 5 learned the game differently from us though instead of playing on the mouse and the screen the ai retrieves the game state directly from dota 2 servers and directly issues commands to the game through dota bot scripting api information such as entity locations cooldowns mana values are provided in the api and are not seen visually by the ai agent technically more information will be passed through the api compared to how we usually play it so in a way it could be slightly disadvantageous competing against humans but even though they are given much more information than us it's not like we can process them if we get that much information anyways but what if an ai plays with the same inputs that we humans are given in this research paper counter-strike deathmatch with large-scale behavioral cloning their goal is to create an ai agent that plays like humans so they train an agent to play cs go just by literally watching humans playing it and it was able to achieve medium term performance with this process in cs go deathmatch and like csgo competitive mode the deathmatch mode reduces the complexity down to the second level as the main objective is reduced to just killing enemy teams so opening a5 on the other hand was giving 170 000 possible inputs to the game while this cs go agent was giving 60 possible inputs ranging from wasd jump shoot reload mouse movements and such the performance itself wasn't that bad and right now you are looking at the footage of the cs go agent playing the game on top left is what the cs go agent sees and in the middle is the actual game footage and there are some interesting things you can observe for instance unlike how cheating programs instantly pin right on top of the enemy's head the aim movement is really smooth this is because first of all cheating programs in fps games obtain enemy player locations and calculate target geometrically so it can instantly aim at the target and second the aim movement on this agent is designed to play like humans so instead of pinning right on top of their location through calculations a swiping motion is created in every frame there are 19 options for the agent to estimate where the target is and move their crosshair towards the enemy through these 19 options while running on 16 frames per second each of the movement choices are half to reduce the choppy movements and create smoother movements creating a 32 fps control another pretty interesting behavior that this agent learned is that it will always attempt spray control in cs go there are different ways to handle an ak-47 the first way is to tap shoot and this resets the recoil every time you shoot so you don't have to worry about the next shot's accuracy the second way is burst fire and this is achieved by firing for two to five consecutive shots and letting the recoil reset and you only need to slightly pull down your aim to hit your target with its slight recoil the third way is to hold down and fire without a break and this is called a spray the recoil of the ak will accumulate and make it pretty hard for someone to control and land all the bullets at the desired spot so spray control is a skill to basically counter the recoil of the gun to focus all the bullets in one single spot back to the cs go ai agent we can see that this spray control behavior was picked up and learned by the agent from the training data however i suspect the correlation between spraying and spray control was a little disconnected as no matter which way of firing the gun the agent will always aimlessly aim downwards to simulate the behavior the agent probably was only able to correlate shooting with spray control instead of types of firing and ways to counter the recoil but at the end of the day this cs go agent is just nowhere near the open ai 5's performance on the result table the above column shows the performance by the cs go agent you can see three levels and one standout section deathmatch easy stands for deathmatch against easy bots with infinite ammo deathmatch medium since we're playing against medium bots with the need of ammo management and deathmatch human means playing against humans also with the need of ammo management the send out section shows an aim training environment where the agent will not die and just need to send sail and shoot at the bots these are all compared against the human baseline were a top 10 percent regular cs go player played in a normal environment the least difference in result is the senso and shoot aim training environment while the largest difference is definitely playing against human players it's obvious to say the cs go agent still has a long way to go compared to that dota 2 ai 99.4 win rate against human players but to my biggest surprise this csgo agent is able to distinguish between the opposing team and the teammates which was pretty unexpected because even i have trouble identifying my teammates in 180 x 80 resolution there are a lot of setups in the cs go ai agent designed to compensate with the inability to scale up largely compared to how open ai does their dota 2 ai things such as resolution size and fps count was limited in the cs go ai research because of the lack of hardwares or datas and also this paper had to choose the behavioral cloning method instead of the classic reinforced learning method due to the lack of cs go api and training data but it's not necessarily a bad thing the agent will be told exactly how to act and behave removing the reward based learning where an agent has to perform trial and error to learn efficiently by itself it's just that it will be limited to how well the demonstrator performs and might be worse as it's only an approximation of the training data so it's a pretty understandable difference when the cs go agent had only 74 hours of experience while open ai 5 had 45 000 years of experience with a different learning method and one researcher can't really compete with the biggest ai research company but imagine if this research is able to scale up to how dota 2 ai was trained it's just pretty exciting to see actual competitive ais that play like humans with the potential of having unique strategies like in dota 2. some of you might not have reached this part of video and are already typing furiously down in the comments about how cheating programs already outperform ai so this is pointless we can't compete with ai's because they have instant reaction time something like that well just let me end this video with this quote from the pro dota 2 player no tell after losing to open ai5 as people it's about being realistic and learning from the brain of the ai not the hydraulic strength that machines have i mean if all you know is to complain about how disadvantaged you are then you won't see what you can learn from it this video is sponsored by 27 stars 27 stars is a london-based development company that creates custom tailored web and mobile applications for individuals or businesses of all sizes they are really experienced and nice to work with and they are also providing an exclusive 10 discount for all of you guys if you choose to work with them all you do is to include my name in the initial email to receive the discount and by working with them you are also indirectly supporting me too which allows me to dedicate more of my time to work on these fun videos thank you so much for watching also a big shout out to andrew and many other patreons and members that support my work through patreon and youtube unfortunately the codes are not released by the author of the cs go agent paper because it can be used by shady people to perform shady practices but be sure to check out their official paper as there are a lot of interesting observations that were not included in this video follow my twitter if you haven't and i'll see you in the next one