Reinforcement Learning: The Next Frontier of Gaming with Danny Lange - #24

Key Takeaways

[Music] hello and welcome to another episode of twiml talk the podcast where I interview interesting people doing interesting things in machine learning and artificial intelligence I'm your host Sam charington my guest on the show this week is Danny Lang vice president for machine learning and AI at video game technology developer Unity Technologies Danny has well traveled in the world of ML and aai and has had a hand in developing machine learning platforms at companies like uber Amazon and Microsoft in this conversation we cover a bunch of topics including how ML and AI are being used in gaming the importance of reinforcement learning in the future of game development the intersection between Ai and augmented in virtual reality and the next steps in natural character and interaction but before we get to Danny does anyone want to guess what today is well not really today but tomorrow and well unless you hear this on Friday May 19th not even really tomorrow for you in any case what I'm trying to get at is that tomorrow is a very special day for the twiml family yep you guessed it tomorrow May 20th is the oneyear anniversary of this week in machine learning and AI I started this podcast last year as a way to share what I was learning about the field and to be quite honest I really had no idea what I was getting into but what an amazing ride it's been the amount of interest in the podcast has been incredible and I could not be more appreciative of and impressed with the twiml listener Community thank you so much we've been working really hard to be worthy of your attention by continuing to bring you amazing content that you can't find any place else while we'd usually mention our quote contest now which was inspired by the way by twiml listener bethan Noble I'd like to do something special this time around to celebrate our birthday this week we'd like to announce a very special listener appreciation contest to honor the occasion first prize in this contest will be a bronze pass to the upcoming O'Reilly AI conference in New York City valued at $1,800 I'll be there and I'd love to have you there as my guest second prize is a brand new Google home powered by AI you know you can listen to the podcast on these right and of course every entrant will win a couple of stickers one for you and one for a friend and of course our Everlasting Love and appreciation so how do you enter well it's easy leave us a review on iTunes or comment on this week's show notes page that'll be at twiml a.com talk sl24 telling us about your favorite twiml experience the things you've learned your favorite moments on the show Etc it's that easy we'll select the winners on June 1st and announce the winners on the next show we'll have an alternate prize available for first place winners who know they can't attend the O'Reilly event which will be of similar value as the second place prize but the details here are TBD all the contest details will be available on the show notes page so click right over or better yet jump into your podcast app right now and leave us a review and wish us happy birthday one more quick note before we get to Danny the future of data Summit was earlier this week and while it was really an amazing event the speakers were just outstanding obviously most of you couldn't make it to the event so I'm working with the interupt conference to see if I can publish the session recording someplace and I'll let you know if I can okay and now on to our show hello everyone I am on the line with Danny Lang Dany is vice president for AI and machine learning at Unity Technologies and I'm super excited to have him on as a guest uh Danny uh why don't you say hi to the audience hey there and thanks for having me absolutely absolutely so uh why don't we start by having you tell us a little bit about your background you have quite an impressive background in this field having built platforms at you know pretty much all of the places that are or many of the places that are doing very cool things with ML and AI why don't you tell us a little bit about that sure yeah I I joined uh Unity technologist uh about 3 months ago uh I saw gaming and and and and and VR and AR as as a huge uh opportunity uh to go in and and change it through Ai and machine learning um but I think uh we'll talk more about that in a moment but but before joining joining Unity Technologies I was head of machine learning at Uber which is a complete different business yeah uh at Uber uh we used machine learning basically all over the company uh there were three really big areas uh one was uh drivers Riders and trips where we use machine learning to to give you a better experience both as a driver as an a rider uh and then there's the whole map business of building maps for the UN for for the Uber drivers uhh uh lot of machine learning there to to basically convert uh Street side imagery and and other data into Maps automatically and then the last area is is of course the famous C course uh was involved in that effort as well uh a lot of machine learning there of course yeah and before Uber was uh general manager for machine learning at Amazon and it's uh it's also very very uh interesting to see how how Amazon really understood to use machine learning all corners of the company I ran uh the elastic machine learning team which was essentially a a platform form uh that other teams could use uh to enable machine learning in their specific business areas so we had you know lots of teams using machine learning that way and before that I was at at at Microsoft I was also working on machine learning there uh so it goes on and on uh many years ago I I spent 8 years in two different startups where we were trying to build uh virtual assistance the ones that we know today as Alexa and Siri we uh tried to build them at that time uh for for the early mobile phones okay and uh that was that was pretty tough because the speech recognition at that time maybe had an accuracy of 80% uh today what is it 95 so those were hard hard times to do to do to do AI because it didn't seem very smart we've made a lot of progress oh yeah okay and so how did you get into uh machine learning in the first place did you study it in school no early on this is very long so long time ago I don't want to talk about it but I I was uh I I I got in when I was at IBM research my key area was autonomous agents so okay it was a lot of the vision of having software that would make decisions on its own you would let it lose I developed what we called mobile agents and it would basically software objects that could move over the internet and and adapt to local environments and things like that but it was all sort of Fairly uh engineering heavy engineering heavy development yeah we designed these we implemented them and if you had a bug in there it would get stuck on another computer not move on and things like that so I got really interested in how you could you could make these agents more resilient and that's sort of where we're headed today with with machine learning with Concepts like reinforcement learning and and and other technologies that uh allow these objects to learn and adopt adapt to the environment they okay uh well that's that is a really interesting topic and one that we won't uh dive into right now um where I really wanted to go with this is you know we've talked about a lot of different applications of machine learning on this podcast but we've never talked about gaming and so I'm super excited to have you on to really dive into how ML and AI are being used in gaming environments right now what's the best way to start that discussion yeah let's start out by by by just acknowledging that whether it's at Microsoft or Amazon or Uber it's the same kind of technologies that we're using it's the same algorithms the same Concepts yeah and you would be surprised that when you move into gaming uh we again are going to use the same Technologies to to bring uh uh more Automation in so to have the games uh uh optimize have the games being interacting more closely with you it's in many ways the same approach the same technique as when we at Amazon try to basically sell you more books yeah right it's this idea that the systems will learn from you and they will adopt is this idea that no developer can sit down and code you know 100 lines of code and then it knows exactly how to get you to buy a book it's sort of the same concept that we want to bring to the game world uh so there are I guess I would describe that world in two broad categories there's uh machine learning and AI that happens in Game and then there's you know all the stuff that many other companies have to deal with you know in terms of making people more likely to play the game and um you know optimizing selling the game and things like that uh are you involved in both sides of the of both of those categories at Unity exactly uh my job is really two jobs so one part of my job is to make sure that uh our many many thousand of game developers can monetize their games that they can make money on their games so that they can develop more games yeah so we uh support and uh advertisement infrastructure we use machine learning to optimize the ads which is essentially you know bringing uh a gameer from one game to another game and the creator of that game then will make some money from that yeah okay and that's a classical uh monetization through ads structure there's also inapp purchases where you will spend money within the game running campaigns and promotions all that is what I would call sort of very classical machine learning and it's super important for the ecosystem uh to be able to to make a living from making games yeah it's very important for our developers that's only half of the story the other half of the story is really uh when it comes to the the creation of games mhm and that's where AI in the gaming World means something slightly different from when I'm using the word AI yeah okay so a lot of Concepts in the in the gaming world is AI means having systems that look and behave in a natural way right it's rule based systems that allow an NPC and and a non-player character or a computer agent to move around in a game and it looks sort of reasonable you know sensible what it's doing but a lot of that is either hotwired or coded through a set of rules yeah let's actually if I can press the pause button there uh this is a question that I've had asked uh before we've had you know in in games going back to you know my Atari 2600 right you have the ability to play against the computer uh is that Ai and um you were just kind of getting at this with the rule based and others and what's the difference between that and you know where we're trying to get to today yeah that's actually a good question it is it is not what I understand by AI but it has elements of it yeah it implements Behavior through rules or or really you know deter istic hardwired code it implements behavior that may emulate uh a human player yeah but want to take that a step further yeah uh that's not the way that companies like Amazon or Uber works yeah when they uh try to to to sell you more products or they try to get a car faster to you um that is basically done through the use of data yeah right that's where we want to move the game world we want to make sure that games uh learn from previous plays and become more and more natural in the in the interaction with the player yeah and that's where where Concepts like machine learning come in yeah I I can give some examples uh of um a particular branch of machine learning which is called reinforcement learning okay and that's essentially uh where the comp computer has this ability to to randomly uh move a character around and and and learn from that that what happens to that character yeah so if you tell the computer that character should not get killed it should be a void uh getting shot by the human player then the character will over time you know basically learn behavior and strategies to avoid void getting shot yeah so we basically say the character has a rewards function which is you get punished if you get shot or hit and you get a plus point you know for every minute you survive yeah MH so with with the reason developments and deep learning uh these agents are able to learn some really complex strategies on their own mhm that make them very very appealing to play against yeah m and so if I'm a player of a game that's you know using uh an AI based agent that I'm playing either with or against how will that feel different to me than you know what I've seen uh with traditional um you know automated opponents or or players yeah the the traditional systems are what we what we could sort of call very shallow they Implement very simple functions uh whereas with with uh this concept of deep learning we're able to implement or the system is able to to represent very very complex nonlinear functions so let me give some examples uh when uh uh when deep mind uh introduced Alpha go um it it it you know it did beat the world champion in go and the way I did that was basically by you know playing some moves that initially looked really harmless that did not get the opponents the human opponents attention ER but certainly in very few moves set the human opponent up for failure yeah uhhuh so uh we have seen another example of uh poker playing using deep uh reinforcement nets for um for poker playing where the system will learn to bloth so the big big difference is that these systems and I use the term deep here they're deep in the sense that they have a very good memory of the events way back that leads eventually leads to great success so they become more strategic than we're used to in some cases they become more strategic than any human player would be yeah so they ready to run the risk like in poker they actually run the risk of losing by bluffing because they know that there is a bigger benefit out there and it's and not because they were programmed to Bluff every so often but rather because they've discovered the strategy through the training process that's accurate spot on and that's the that's the new thing that's what we're able to do today and and that really changes the a part of game development to become more about defining the objective or the the success criteria for that NPC because let's just say that we Define it as though shall not get shot yeah then the agent will really you know be really good at hiding yeah you and you will get really good at it yeah or just getting away in time ever every time you try to find it yeah that's not fun yeah so what you have to do is you have to set up some some some criteria for that NPC to that eventually will allow it to develop some subtle and complex strategies that you cannot predict per say but we know will lead to some very very appealing uh behavior that will give you know great enjoyment to play against because they are maybe hard to predict maybe you you constantly think you're about to have this agent and then they fool you and then it becomes very entertaining yeah and that's what we have seen that these very deep these deep Nets uh uh through reinforcement learning can really build up these very complex strategies that that that that sometimes I saw a quote from One poker player who basically stated that if he didn't know better he would he he would cat iiz that as a really like a human spirit in in that game play yeah uhhuh yeah so the I think the popular examples of um or at least the this you know the celebrated AI successes have been around these strategy board games like go and chess uh to what extent are games like you know your Xbox games like Halo or Call of Duty to what extents are the various agents in those games powered by something that you would think of as AI today uh today they are powered by what we would call Classic Game AI so it's it's in uh in in most most of the cases that I'm aware of it's it's very complex rule-based systems in some cases way way over a thousand independent rules that have been created by the developers to basically cover a lot of different situations uh it's very complex systems uh but we we saw a similar approach in uh in antivirus uh programs for a long time they developed basically databases of rules that would identify you know components or elements of Ros software and today all these systems have moved on and are actually using a lot of them are using uh deep learning uh uh for for training on on on known viruses to detect you know new viruses and are you aware of any particular uh efforts or results that have been maybe published somewhere uh on in terms of introducing AI into these types of games no not not in any commercial games uh we would of course you know from Unity technology side we would uh that's one of our objectives is to to to offer that uh sometime in the future uh there's a lot of research going on uh so if we go to Academia go to the research Community there's a great interest in uh in uh in start working on this uh the the appeal is of course that if I can do this if I can get an NPC in a game to to act in a way reasonable and intelligent uh and constructive way uh then you know eventually you know maybe real robots could behave that way too right right so how do you as Unity how do you uh expect this to evolve and are there a set of services that you uh are hoping to deliver to game developers um how do you think this evolves over time uh to so that we'll see it more in games yeah it's it's my my hope that we can make some of these uh uh algorithms uh available as Services uh essentially to enable a game developer who is not an AI or newal network or ml guy you know some DB specialist but basically a game developer make a game developer having a tool box of of of of AI uh services that they can use to train a character uh and in that sense sort of gradually enable AI in game development we have to understand that this is this is the livelihood of of of you know our developers this is how how they they make money and that's how they live yeah they develop games so what we want to do is we want to give them these tools that allow them to create maybe uh you know maybe make it more efficient they can G create some games faster they can create games that are more entertaining uh Etc yeah uh so we want to make that available so that it can sort of gradually be be uh be adopted and we can also learn from the use nobody has really done this very much yet yeah so we don't know yet how how it's going to work in practice in some ways there's a it strikes me that there's a precedent for this and that game developers aren't typically physicists either but it's very common for them to work with these physics engines nowadays to develop their games no that's correct uh we we have um we have actually done some uh studies where we using reinforcement learning to have characters learn how to move in a world where there's gravity and friction yeah uh and uh and inertia and it it's kind of interesting to see how how a character can learn uh you know to to walk or run or jump uh without knowing it from the outset but by basically attempting you know T of thousands of times you know and over time you will figure out the pattern of movement uh to jump for instance uh and in that sense you know we are we actually using the fact that that we can we can simulate gravity simulate friction and inertia in the physics engine okay um yeah one question that I've had about the application of reinforcement learning to games is I guess when when we think about machine learning in general and I don't believe that reinforcement learning uh is a common exception to this you've got kind of your two separate stages you've got your training stage and your inference stage um but in a game environment I imagine part of the appeal of AI is so that the the agents in the game can learn from the game as it's happening right so not during a training phase but during the the actual playing Phase uh how is how does that um you know is that we've talked about on the show Active Learning which may be a part of this but is there a specific name for that and uh how is it commonly done today yeah um you you're absolutely right yeah that that's where I would love to go I I would love to go let let me give some crazy crazy examples yet but imagine that uh say that you are playing your Pokemon Pokémon go yeah when you have to train your character yeah today the training of that character is just it's not really anything yeah you just click a few times and and that's how you train it yeah but imagine that you are actually training that character to battle and you could train it for half an hour one hour and that you are actually teaching and reinforcement algorithm behind the character yeah in real time and if you're really good at training it it may beat the some other uh person's character yeah who was not that good at training it that's one thing the other thing is that if the game can learn through game play then it can learn uh about you and just like Amazon and Netflix uh they basically tailor everything you see to right to you I mean if you go to Netflix what you see is different from what I see yeah it's tailored to you what if every game in that sense was tailored to you yeah if if the game knows that you're only happy if you progress fast through the levels then let's make it easy to progress yeah but if the game knows that I'm sort of more rigorous and I I want you know to fight hard from get to level to level then um then that's a different different approach yeah what if games could adapt oh that's really interesting yeah I'd be happy if the games could know that uh you know whenever I'm playing Call of Duty I'm hiding in this particular corner and make it a little bit less easy for me to just camp out there and snipe on people exactly yes you mentioned that one of the one of the motivations for you joining Unity was uh what was going on in terms of augmented and virtual reality can you talk a little bit about that and how where you see the intersections between arvr and AI being yeah now if you think about a game as you know it today being it's it's very controlled environment yeah I I can I can hot code the game I can add AI to the game but it's a very controlled environment yeah right when we start moving into VR and AR we start having you know a lot more flexibility so so let's take take VR a bit yeah now I I I emerge into a game I can now you know move around in the game and I can you know I can pet the characters in the game I can touch them and when I do that the bar moves up for what I consider naturalness yeah I I may play a game on a phone and I can see the characters are not you know they are characters and they are a bit artificial in their movements yeah when I put on my golf go uh I I want to get immersed into this this new world yeah and things have to act in a very natural way with me otherwise the illusion breaks and it's not really fun and take that a step further in in in augmented reality the the game or should we call it the app because it can be way more than games yeah let's go the app yeah needs to sort of uh connect between myself and the surroundings yeah and those surroundings can now be anything yeah I I don't know you you use this app I don't know where you're going to use it I don't know what's going to be there so you see the bar goes up constantly up where you can no longer imagine you know a thousand lines of of uh code being the answer to everything yeah so that's where you need to have ai really come in yeah and that that is is is of course a an area where we can we can make a big change through AI clearly I I think that makes a lot of sense the user expectation is dramatically increased because you want this environment to be immersive and engaging and feel lifelike and so it only makes sense that you'd have you're interacting with intelligent things if they're you know beings do you expect the the AI techniques that we use to do this to you know be the same kind of things that we're applying every place else or the are there going to be a new set of techniques that come into play when we start talking arvr I think it's going to initially it's going to be some of the same Technologies but I want to say one thing here which is when we do talk about this of VR and AR that we are able to push the boundary of AI much much further here and much faster yeah if I'm building a self-driving car I I have to be very careful how far I push the boundaries of a a AI yeah because it's it's it's a two 2,000 pound or more object that may actually kill people if I'm not you know understanding what I'm doing as an machine learning on AI expert yeah uh so you need to really put a lot of boundaries in uh when you build an industrial robot it's the same yeah it needs to build a be a car but it it it it it it it should not you know hit the operator around it it so it has there's a lot of hard wiring taking place when you move to VR and AR I'm like how you know how how bad can it be what harm can you make yeah so you can really push the boundary there yet you create a a a a complete you know believable world yeah at the same time yeah so so you give people this experience and you can push the boundaries of AI to its very limits and if you go over people take the goggles off and say n that was really bad you know that was not believable yeah right yeah and that attracts me a lot to this space here because when you sell books or you get taxes around you are to a great extent limited by the physical world right yeah right now we can we can really play tricks with the physical world and if those tricks they fail we didn't really kill anyone have you seen any interesting implementations or early results in marrying Ai and arvr we we have done some experiments uh I think that this is uh this is still uh emerging technology uh there's a lot of things around VR and AR that has not yet really being tried out and not yet fully understood uh but I think that very quickly uh creators of of games and apps in AR and VR are realizing that they're going to have to step it up a bit because people expect that natural interaction yeah so in in in in many VR apps and today you can you can pick up an object and you can maybe throw it yeah but you would love that same interaction with other virtual characters or nonperson characters in in in those games yeah right and you really like to interact with them and you want to interact with them in your way and they should be able to respond in a natural and not you know some uncanny way mhm and I think we still have some way to go there right right yeah I think that may be a bit of summary of that one bucket which is the in-game experience and inv VR AR experience is it it's just super early and you guys are experimenting as are a lot of people but um you know we're still far from you know Call of Duty X whatever the next version is being a fully AI enabled game yeah that's for sure the part we're trying to play there is to enable the basic Technologies so that our game developers can start playing with it and sort of evolve it and help us understand what works and what does not work mhh and so how about in the other bucket um where you're uh applying ML and AI to you know more for for from a business perspective what are some of the results that you've been able to uh achieve there yeah so so by basically on understanding what kind of games people play just like what kind of movies you watch and what kind of books you read you understand what kind of games you play you you're able to uh uh pinpoint much better recommendations or ads uh for other games yeah so you will you will if you always play single shooter games uh we you know probably would be best off showing you some trailers for other single shooter games yeah right but there are Technologies in there that Amazon that uh Netflix has been very good at technologies that basically start exploring uh uh your your interests even exploring the things that you maybe don't even know yet that you're actually interested in so so basically using machine learning to better get to know you and and your game preferences and make sure that you can discover those games because that there are so many many many games in in the App Store yeah or in Google Play that there's so many games that you can't possibly find all the games that you actually really like yeah so we will basically learn over time uh what your preferences are and find those games for you what have you found challenging in applying these Technologies at unity and for that matter at previous organizations what you know what are the challenge patterns that you come up with when you're trying to introduce this or mature it in a given organization yeah yeah that whether that is Amazon Uber or or or Unity uh technologies that it's the same all the way around and that is that you should not need to be a machine learning expert or an AI expert uh to accomplish this you should be able to say my problem is selling more shoes or selling more books or getting taxes on time therefore uh I should be able to use machine learning services to get there right and that's how I thought about it at Amazon by basically creating as a platform we also launched uh the Amazon machine Learning System we launched that on on Amazon web services for the public to use mhm and that is very powerful because now you get people with real business problems people with real incentives to improve their business using these Technologies which are way Superior to any attempt of trying to to write you know a thousand lines of C++ code or whatever to solve the problem yeah basically based on that data they can train computers to do a better job yeah so in in all these organizations was a matter of making it available in a meaningful way and that's what I'm I'm doing at Unity to is to make sure that game developers do not have to have a PhD in machine learning to benefit from it what lessons have you learned in building these machine learning platforms at Amazon Uber and now Unity the lesson the lessons this's one big lesson learned which is that it just works really really well uhhuh so make it just work it it but but no M machine learning does work really well it does it does provide results period and uh uh there may be a lot of talk about it there may be a lot of talk about deep learning uh some people call it hype I'm like yep you can call it hype you can call it whatever you want but the the the the proof is there that we are able to improve business where we move from a concept of trying to analyze and investigate and Implement to a world where we basically use the data to to improve the business through learning and whether that is selling books driving taxes or developing and playing and selling games uhhuh machine learning is is it really works you've um been talking to different groups about the notion of bringing machine learning to every corner of your business um and to me that suggests you know not just the obvious places um but maybe some interesting places do you have any uh stories or examples of you know interesting Corners that you have been able to bring machine learning and you you know where you found you interesting or surprising results yeah um I I would say that uh you often run into business problems uh uh if you're selling shoes you're one of your key business problems is that people get get a shoe online order it online they get a shoe and doesn't fit yeahh the machine learning is one of these wonderful things where all the return data all the shoes that did fit and all the shoes that did not fit if you use that data you can start predicting how different uh shoes shoe brands you know do they run large do they run small or they spot on you can even start uh predicting what what shoe size a given uh customer has yeah and you can basically use that to give a better shoe purchasing experience to me that's kind of a surprise yeah I I didn't really think about shoes but that's when you enable machine learning uh to into a non-machine learning environment you will see business problems popping up another one is Imagine uh I've seen this being used uh I won't be specific but where machine learning is used within HR to to basically uh complement or one day maybe even replace you know performance reviews and things like that yeah MH uh to predict uh to analyze resumés and and and predict you know which uh candidates may be good and strong uh future employees right um so there are areas sort of to me uh because I've always been on the platform side of this it's it's it's I I get surprised uh almost if not daily but then close close to daily um on on what people are using this for yeah and and that is really what matters is it matters you know let's enable a lot of people to use this technology and so in order to to to make that Vision real how do you build the right relationship between the you know the platform people the data science people the business people so that you can kind of build capacity as an organization that uses machine learning effectively yeah you need to you need to avoid making this a um a very custom effort for every single team and for every single use that's what it traditionally you know that's where it came from yeah you would have a team that would have a business need you would hire data scientist you would hire machine learning Engineers you would Implement proprietary algorithms and uh you would maintain those and then you would need to hire more people as your business scales Etc yeah um and and one thing that um I did at Uber was to build a machine learning stack from the ground up entirely based on open source and Define it build it and Define it as a platform yeah so what I find over the years is that a lot of Team come and say I need this feature I need that feature this is very important this is very important for me I need this other feature which is even more important for me well that's the wrong mindset because what they need to do is they need to deploy a lot of machine learning uh models across their business rather than just fixating on one particular one yeah and I have often seen this in say in in fraud detection teams they would have some Fair custom algorithms detecting you know certain kind of Fraud and I sort of compare it to you know you have five box on your front door but you leave all the windows open yeah so these very customized Solutions uh I think should be something of the past because using open source using standardized algorithms is so much more efficient so rather than one Magic model deploy 10 or 50 or 100 different models right run your business can you speak at all to the specific open- Source tools and projects that you use to build that platform at Uber absolutely it's it's it's all public there are slides around if you Google it um but it's essentially a a Hadoop uh hdfs Hadoop spark things like ml lip uh XD boost an open source uh very very versatile open source uh algorithm from University of Washington uh things like Cafe tensorflow uh um kfka for streaming um Cassandra for uh for for structured data it's it's you can get really really far with all those tools uh combined uh without any need for uh for really deep uh customization and so the platform you built was um something that made all of these available and did you put some kind of layer on top of it to you know for visualization or automation or how did you pull all of these disparate tools together yeah uh both at at at uh Amazon and Uber the the the key contribution was essentially uh a uh defining you know a really solid SDK so people could programmatically integrate with that system okay uh a uh nice and easy to ous graphical user interface a web based uh browser based uh interface uh again because we want uh wide range of users to to to not feel that the barrier of Entry is too high yeah uh and then so so really the user interface there DEC side that's how to access the systems and then at the other end of it you need to have the system being able to easily access the data of the company so you need to integrate on the data side into the data repositories of the company whether there's existing data warehousing data Lakes uh you know relational databases Etc yeah so so the stack is really uh is really something I I I would say is not the stack itself is not as important as long as it works and runs but it's sort of that integration on both the front end and the back end into to the business that's important because that lowers the barrier of Entry it makes it easy for you to build a model if you don't have to track down uh data from outside your team mhm um on the front end of that was the primary user experience like a IPython notebook or Jupiter notebook rather uh or um were there different ways that Developers access the system yeah uh there was a a a a total Custom Custom user interface there was also act you could also basically use uh things like Jupiter notebook as long as the system has a python SDK you can do that integration very quickly yeah and you now run your you can run your your model building jobs from Jupiter you can visualize stuff you can deploy from Jupiter yeah uh and so I I personally I love I love Jupiter notebook I think is an incredible tool um a custom user interface fine too but most of all act is is the the sdks in multiple languages are important because that allows uh bagend Engineers to to easily integrate machine learning into their offerings yeah so if you take the mobile app at the mobile app at Uber uh when you open the app up up it suggests two one or two or three likely destinations that you normally go to that piece that's that that was actually a piece of machine learning right there yeah using your data to build models that predict where you're going to go you know on a given day of the week at given time interesting uh when we spoke previously you talked a little bit about your vision for really fully autonomous corporations what does that mean and uh what do you envision there yeah if you follow me on thinking that we can use really deep reinforcement learning to train an NPC or an goal player poker player yeah if we can train a computer to use really deep and complex strategies including bluffing um if we can train it to do that then we can probably also train an agent to say sell books or sell movies yeah and become better and better at it yeah I mean like in the end it should become the best book selling agent in the world yeah and you can take that further and say well uh the point for instance let's let's just take an example not only which books should I try to sell but what should the prices be yeah imagine that you use reinforcement learning to figure out what is the right price to sell stuff at yeah and I what is the right time to order books at so that we don't run out of book for Logistics reasons yeah when should we order the parts to build you know our Xbox our our Kindle our Hardware yeah if you look at a lot of uh problems that humans are are solving in corporations today I think many of those problems can be learned and solved by by a computer if if you take re reinforcement learning deep reinforcement learning as we know it today as a starting point and basically just think a little out there um why not right right yeah I think what's interesting about that to me is that we often think about this uh AGI artificial general intelligence as this kind of one Uber AI that you know is just very humanlike and can reason and um you know do everything uh but you're describing a you know something that that could create tremendous value that is you know much less ambitious right it's a collection of uh of AIS that are more purpose-built uh and we should be able to get there you know a lot quicker yeah I I I I'm kind of struggling with the whole AGI debate sort of maybe you know plus one or plus two out there for me uh but I do think that that there is an interesting perspective in if we envision a world where we enforcement learning I'm using that as sort of a very general term here yeah but there will you know be many different variations of it but we do Envision a world where a company like Amazon use that not you know just to selling books not just in 10 places but they use it 5,000 parts of their company yeah they use it everywhere yeah right and you start having these systems interacting um you know at one point uh when you add it all up you get more than the sum yeah M mhm because now you're going to get the effect of I'm am really good at selling I'm really good at finding the right price point to maximize Revenue I'm really good at understanding what people want I'm really good at understanding ordering all the parts and components in time so that I never run out etc etc and suddenly you have sort of a whole product pipeline happening yeah um with very limited uh human intervention yeah is that AGI I don't think people think of that but is that looking like a super intelligent Corporation oh yeah yeah it strikes me that there's a whole field of study and research that would need to happen to allow us to manage this thing like apply it's almost the application of traditional control systems work to these independent AI systems like hysteresis so you don't you know over or under order or order too aggressively it's just a whole layer that needs to happen there that we're clearly you not there yet yeah I I I totally agree with that observation and I also think that and and and we start seeing this in in in in in some papers coming out I do see a sh a shift uh in in some research that now goes more towards researching the strategy around teaching computers yeah so basically examples are like if I give a reinforcement learning algorithm if I give it too hard of a problem to begin with it's not going to solve it yeah but if I sort of give it you know increasingly difficult problems to solve it gets really good at it yeah so that's kind of learning strategies and uh so we're not talking about we're not talking about the technology here we're not talking about you know this particular structure of a deep newal net or something like that which is talking about how do I best teach a computer yeah and that company is looking at how can I minimize the need for data because data can be you know difficult to get by so way I can I can learn more efficiently so if you think about some of these uh research topics are kind of at a meta level yeah so far we have been so eager to study algorithms uh and methods uh but maybe a lot of the future research is really about what what does a good school for computers look like interesting interesting uh well as we wind down any parting thoughts or words for our audience I think that the audience should look at uh the gaming and the gaming platforms as a fantastic uh lab uh for AI that going to give us a peak into the future because if you think about NPC moving around in a game you know they are really robots yeah but robots in the physical world they take you know easily year to two years to build yeah and they're expensive very expensive lots of uh mechanical engineering yeah but if you think about an NPC in a in in a computer game that's pretty much a robot too uh you can evolve them quickly you can try out different you know Dimensions different make mechanisms uh everything can happen at uh at uh you know surreal speeds yeah compared to the real world so I think that I the way I look at the gaming world uh is Way Beyond gaming I'm looking it as like what is going to be possible in the physical world you know 5 10 years from now well that may be possible in a gaming world next week M that's awesome that's great great all right Danny well thank you very much for joining us it was it was great having you on the show and great chatting with you oh it was a pleasure I enjoyed it thank you all right everyone that's our show for today once again thanks so much for listening and for your continued support don't forget to leave your don't forget to leave your review or comment to enter our one-year anniversary listener appreciation contest the full details can be found along with the notes for this show over at twim ai.com talk sl24 over at twim ai.com talk4 where you'll also find links to Danny and the various resources mentioned in the show you are all amazing and we love you thanks so much for listening and catch you next time