Machine Learning Analysis of Player Behaviour in Tomb Raider: Underworld | AI and Games #31

Key Takeaways

[Music] hi I'm Tommy Thompson this is a eying games and in this video I'm going to be talking about tomb raider specifically Crystal Dynamics 2008 release tomb raider underworld now why would I take a look at underworld itself the third entry of a reboot of Tomb Raider just the shadow of the Tomb Raider the third entry of the second reboot of the franchise has been released now in 2018 well hidden behind this game is the story of one of the first major efforts at analysing player performance in a triple-a title he achieved through large-scale data collection and a little bit of help from artificial intelligence all in an effort to answer the question do players actually play the games we make in the way we expect them to nowadays it's incredibly common for games to adopt some form of analytics a process whereby data is collected in how well the game performs as well as how players behave within it it allows for large-scale games such as fortnight to be able to react to community behavior ruling out updates frequently to address gameplay balance based on player data plus many games particularly those with free-to-play monetization models adopt this process for the purpose of recognizing levels of engagement and an effort to drive more revenue from their most active players this isn't the first time we've seen how large-scale data collection can be used to model aspects of player behavior in games today I've covered two academic projects namely player performance analysis in battlefield 3 as well as status analysis and the cosmetics economy in Team Fortress 2 each of which highlighting how data about players can tell us interesting things that we perhaps wouldn't expect however it can also be used as part of core gameplay a point I covered when exploring the shadow AI in the reboot of Killer Instinct that replicates how human plays to create an AI counterpart player analytics has become increasingly commonplace with the surge of cloud computing and machine learning AI in recent years but conducting this type of work even 10 years ago was a bit of a hurdle the technical infrastructure wasn't in place and arguably the expertise in how to process and handle data at this scale wasn't yes polished as such let's take a look at how this work was achieved in Tomb Raider oh dear that was collected and processed and what they actually learned through this experiment the research in question started back in 2008 and was intent on evaluating her audiences play a game once it has been released in the wild well it's commonplace for focus testing to be conducted during a games development the work was intent on better understanding how argot demographics play in their natural environment this would help identify whether a game is meeting the expectations of both clears and signers but looking how its performing in the wild if you will well it may not seem obvious even conducting this type of research in a games such as Tomb Raider is incredibly valuable even as the series's went through numerous reboots and interpretations the core of Tomb Raider has largely remained consistent lara croft needs to navigate a series of Dettol it's dangerous and often trap written environments typically tombs fight off enemies and solve the odd puzzle hero there before finding and subsequently raiding the treasure vaults deep within despite this each iteration of the franchise and their individual entries rebounds as these elements in different ways and may not yield the desired response from audiences so it's best understand what's working or not for players such that designers can capitalize upon now the project itself was kick-started by game user researcher Alessandro Canosa at the beginning of the project Canosa what for Danish developer IO Interactive the creators of hitman who were still at the time subsidiaries of tomb raider publisher idols interactive who were later named Square Enix Europe in 2009 was a mouthful moving on through this company structure Canosa and intern Crystal Dynamics the acting developers of the Tomb Raider franchise could utilize what is now known as the Square Enix Europe matrix suite an event logging system utilized by a variety of developers under the publisher for recording data about how players are interacting with their games for Tomb Raider that's included reporting when and where players die in the game and time taken to complete parts of each level in an effort to get players data from their natural environment the D are used in all of the research in this video was acquired through xbox live around the fall of 2008 when the game punched the sqe metric suite recorded data from around one and a half million players though not all of it proved useful for the research given it was broken inaccurate or incomplete however tens of thousands of valid player instance data could be found within the database and was subsequently utilized it was at this point can also turn to researchers based at the IT University of Copenhagen critically analytics researcher and das Drakon whose work recently appeared in my overview of AI research in StarCraft there were some immediate avenues that this research could take such as applying the data of around 20,000 players who had valid geospatial metrics meaning that the 3d coordinates of events logged matched against parts of the game map these metrics could be assessed against the actual maps of the game to create visualizations of player death frequency the variety of deaths occurring and how long players spend in specific part of the map utilizing existing geographic information system technology allowed for visualizations such as this one of the valley shelf map from the latter half of the game has helped to identify which parts of the map were proving more challenging than others and isolate areas that may require tweaking well this is impressive it still didn't require any AI to help to run these processes Canosa and rockin alongside noted AI researcher URI Oceana caucus at the time also based ITU decided to explore something more ambitious could a machine learning algorithm trained on this data in order to establish aspects of player behavior to do this they gathered up data from just over 25,000 players who played Tomb Raider underworld in November of 2008 but focused specifically on the 1365 players than their dataset that had completed the game in its entirety to run the experiment the team extracted six major gameplay features from each player the total number of deaths the cause of each death completion time and how frequently they used the help on demand feature an option that allows for hints and solutions to puzzles to be presented for players whilst tangential to the study it does raise some fun facts about this player base for example the SE of 1300 players recorded over 520 days of playtime averaging around 10 hours each and died over 190,000 times but also identified the variety and player competence in the game with the fastest completion time in two hours 51 minutes and the most deaths recorded for any one player being 450 aides interestingly it highlighted that death by falling was the dominant issue for players with 57% of all deaths recorded from Falls whilst just under 29 percent came from fighting non player characters and the remainder being environmental issues such as drowning traps and fire the team first utilized k-means and wards hierarchical clustering to identify whether this large set of data could be reduced to a smaller and more manageable space that identified trends of behavior this first phase identified around three to five clusters that could exist and as such a second analysis was conducted using a variant of neural networks known as emergent self-organizing maps the results behind this are pretty wild the Isom identified four major clusters of players that exist within the dataset runners solvers veterans and pacifists each having a specific combination of traits in their data that told us something about how they play Tomb Raider first up runners have very fast completion times with many of them having similar completion times to one another plus they die often with many instances coming from opponents in the environment one area that is quite varied within this cluster is the use of the puzzle help feature with some opting to use a lot and others hardly ever meanwhile solvers are the exact opposite they seldom ever ask for hints to puzzles and solve puzzles quickly however they have very long completion times as well as few deaths by NPC or environment but they actually die a lot from falling and suggests that solvers adopt a much slower methodical and careful approach when possible thirdly pacifists die mostly from opponents but only have slightly below average completion times and minimum help requests and lastly veterans appear to know what they're doing they play through the game quickly though typically not as fast as runners and while they seldom die still most of their deaths occur thanks to Falls an environment with his body of work complete at left an open question what if using this data you could predict how a given player would behave in the future how quickly can we establish that a player may adhere to one of these archetypes or perhaps a more useful piece of information about player engagement are they going to stop playing because it's too frustrating for them are they even going to finish the game and if so how long is it going to take them a follow-up project by Tobias maehlman alongside Drakon Kenosha Jana caucus as well as you lean to galius use the existing dataset but both expanded and contracted the set of data being considered first of all the number of players was increased to ten thousand of which six thousand four hundred and thirty were usable but the actual data from each player was no longer constrained by whether they actually completed the game this resulted in three data sets the first containing two thousand five hundred and sixty-one players that only completed the first level the second containing three thousand five hundred and seventeen players who completed at least levels one and two and the final one containing the 1732 players from the dais a who completed the entire game sometime between December 2008 and January 2009 in addition to extracting the play time the total deaths this time a staggering nine hundred and sixty one thousand causes of death and the use of the help on demand the team also pulled features such as how many artifacts and treasures were collected as well as settings being modified in the options menu Tomb Raider underworld permits players to customize difficulty by tweaking amyl enemy hit points Lara's health and the use of saving grabs interestingly fifteen thousand three hundred and seventeen changes were made to customize the gameplay experience but only one thousand seven hundred and forty players and the dataset actually used it this time around the team sought to Train two predictor systems the first to predict how many levels our player may complete and the second to estimate the total time it would take said player to complete the entire game this was being based solely on data about the performances in level 1 the Mediterranean Sea and level 2 coastal Thailand this was trained using the Weka machine learning platform from the University of Waikato in New Zealand which contains a variety of analytical and machine learning algorithms that are ideal for these purposes most far from perfect the results are fascinating the results vary significantly but numerous algorithms perform better than the predicted baseline that the authors estimated however what was really interesting was analysis conducted by the authors on the rep tree algorithm that they tested against this algorithm could predict the final level with 48.5% accuracy using just level one data but only used one rule how long the player spends in the opening area of the C at the start of the Mediterranean mission this could achieve almost 50 percent accuracy based solely on how long you spend floating around at the beginning in addition when expanding the dataset to levels 1 & 2 its accuracy improved to seventy six point seven percent but only considered one location from level two as well as the total rewards collected in level two further analysis of this phenomena helped establish that of fifty-five features recorded in the first two levels four in particular were deemed more significant the time spent on the C top in level 1 the Norse hall in level 2 alongside the amount of help required and rewards gathered in level two despite this the resulting output for predicting time didn't yield any major improvements with the relative absolute error still being significantly high but hey one or two ain't bad right lastly a final body of work was published in 2013 most Canosa and Dragon were both based at Northeastern University in Boston having assessed what types of groups exist within the data and what features helped identify players within them this final experiment sought to assess well these groups evolve or change during the game using the same set of features as the second experiment this project explored data from 62,000 players who completed at least a significant portion of the game from a collection of around 200 thousand from the same existing say gathered between December 2008 and January 2000 named this time using a form of archetypal analysis the researchers analyzed two particular aspects of the data first how players that completed the game which was around 16% of the 62,000 spent their time in levels and second analyzing how the features being recorded for each player changed as they progressed between levels this resulted in six archetypes being established from this data with players migrating between archetypes during their play time while each archetype shows a gradual increase in completion time given the levels are getting harder they vary in the actual time taken showing variation in player skill in addition each level shows a different distribution of player clusters highlighting how some become more prominent others in specific levels of the game while other clusters only exist in very specific levels when cross-referenced against the original player clustering work they found for recurring clusters that should strong similarities to the veterans pacifists and solvers in the original experiment veterans matched up with adrenaline rewards shown here in light green a cluster of players that complete the game fast make few adjustments and heavily use the adrenaline feature in the game death reward environment a less consistent cluster that's captured by the dark orange platinum and black segments matched up with pacifists due to an ability to collect a lot of artifacts and dying from environmental causes and lastly time reward in yellow which should strong similarities to the solver profile where players generally perform well but are very slow to complete the game and collect a lot of rewards in each map woof that was a lot research wasn't it well tomb raider underworld is largely forgotten in the ongoing Adventures of lara croft the impact this game had on the emerging game analytics field and AI research as well is hard to ignore and much of the contemporary research literature and game analytics tomb raider underworld is celebrated as one of the seminal bodies of work that established what we can learn from even the smallest amounts of data and pave the way for more exciting projects in the future some of which i already covered here on the channel it's fair to say game analytics are only going to increase in scope moving forward so who knows maybe we'll find out in a couple of years whether players are feeling better and the more recent entries of the franchise hey folks thanks for watching this video on tomb raider underworld this has been on my to-do list for quite some time now so I'm really happy to get it out there and let everyone find out about this really cool body of research that happened quite a few years ago for those of you who are really interested I've linked to all the original research papers in the description so you can go ahead and check them out yourself in the next video I'm returning to the world of Far Cry and specifically taking a look at some of the weird tweaks that had to happen under the hood in order to ensure that the animal companions in Far Cry primal stop running off of cliffs