Bridging traditional "onprem" game development with cloud workflows

Key Takeaways

[Applause] hi everyone um i hope that you your families and your communities are all doing great at this time um thank you for joining mix this year and joining this session in particular so today we're going to talk about bridging traditional game development with cloud workflows so before we actually dig into it i'm a big dreamer and i'd like for all of us to share a dream basically let's dream together that triple a games will be accessible to everyone using only powered battery-powered devices through the internet oh wait that's already happening right so what could be the next big dream well it could be that we build those games using the same id only battery-powered devices no big infrastructure very something that is easy to access and that lets us build games executables cook data in matter of seconds instead of hours that could be an interesting dream but where are we now as an industry right now major players that were previously outside of the game industry recently joined and they announced their own game streaming platform on the other end of this industry studios are all embracing cloud computing whether it's for game as a service or to host their game servers or even to build new pipelines on the cloud yet many engines pipelines and workflow workflows we have built uh have been built with on-prem hardware and on-prem workflows and we still focus a lot on it so where are we adding then um given that dream and that state we might be adding to cloud native games engine and pipelines at least that's what i dream of so i'm gabe uh i joined stadia games and entertainment late into 2019 before that i worked at ubisoft splash damage and other companies i also tried my own company outside of games too i'm a technical architect which is a nice way to say i'm a software engineer i like combat sports and cats and i worked on different types of games from sports to open worlds to for special and shooters through online games with cloud without cloud so before we dig into the challenges that uh we're gonna face to bridge our workflows to cloud native workflows we have to define what cloud native is and wikipedia can help us defining cloud native computing as an approach in software development that utilizes cloud computing to build run scalable applications in modern dynamic environment such as public private and hybrid cloud that's a mouthful we have to start by defining what cloud computing is and what are those dynamic environments cloud computing is the on demand usage of computer system resources basically your cpu your ram your gpu storage and networks can be extended you ask them on demand you get them and you can release them with higher flexibility but cloud computing is also about managed services with high availability you don't have to care how to run your next stack you can just deploy it and trust that a lot of the infrastructure will be taken off your shoulders um that definitions also brings scalable applications in in the picture and when we talk about cloud a lot of us think about horizontal scaling oh that's cool we're going to add more machines or more processes to solve the problem and serve a bigger population well i would like to also point out that vertical scaling that we all experience either when we changed our on-premise hardware to get new ssds or bigger cpus or when we changed console generation like vertical scaling happened at that point we know about it but in the cloud vertical scanning has different limits uh way more cpu than what you're used to and more ram that you used to as well before we continue on what cloud is and what it brings i would like to picture it for game developers we've been used to do some things for a long time but cloud isn't that different from what we've been doing remember the transition a couple years ago to multi-core architectures the transition to the ps3 where we used the ppu to send off some data to spu so that those spus could do compute and send back results while cloud computing is pretty close to that right you're going to send data off that data is going to land somewhere it's going to be computed by something and the results are fl are flowing back either to your game servers or to your workstations so we already done that the other important aspect that game developers tend to care about is storage when we build games we try to optimize our access patterns for data and that shows in our workflows on our day-to-day pipelines we think about registers we think about cpu caches we think about ram and this extends all the way to local storage two programmers might add to that on-premise storage network attached storage to distribute builds uh maybe even on-premise cloud storage while moving to cloud we're gonna deepen that stack we're gonna add to that edge storage like the one that is used for cloud delivery networks as well as cloud storage the one that is used in data centers um game developers tend to talk about access speed and with that comes storage scarcity basically the faster you go the more expensive it is so the less you have well in cloud we tend to talk about latency and storage size and you basically have to reverse that stack easy we know about all of that so what does the cloud look like i introduced two concepts cloud storage and edge storage so let me talk about uh what what it looks like in reality a lot of people think that you basically have your device it's connected to your isp and that's it you're connected to thousands of data centers it's not exactly true from the point of view of our workflows you usually have your device that is connected on premise to your networks that on on-premise architecture might have a private cloud with some on-premise storage or build machines on premise compute it's connected to a gateway and that gateway is connecting you to your isp your isp has a presence in an internet point of presence or an internet exchange point usually it's a building where several companies are collocating hardware that way your isp can talk to other isps or other companies such as google amazon or any cdn company in that point of presence we have edge compute and cdn caches so edge storage it is not cheap but it is close to your users and then that point of storage is connected through ultra fast uh networks to data centers and at that point in data centers compute and storage becomes widely available and communication is very fast within zones within the data centers and up to your point of presence basically the slow link is most probably between you and that point of presence so let me walk back a second and and talk about what we're solving again many games are moving toward game as a service uh we're adopting cloud native workflows because it helped us build and push content faster to our users but cloud doesn't offer just that it also helps companies with managing on-premise uh computer resources that is complex so cloud can help you by taking off your shoulder that manage infrastructure ideally you can you can focus on your product the last point i want to to bring is resource capability it is really interesting to see that when we need more build machines it is actually complex to get that when you're on premise it's because you have to go buy them set up them add them and basically if you don't need them the next month where they are still there you paid for it so cloud if you need to produce a bin faster well you can commit more resources to it a bigger machine if you want to produce multiple builds in parallel you can commit multiple machines to it but if you're done with your workflows well you can decommit that you can release your resources and stop paying for for them to me that that that is accessibility that is flexibility and those two advantages are super interesting from a pipeline point of view so now let's talk a bit about the challenges of moving our workflows from on-premise to cloud but before that i have to confess about something before working with cloud i didn't have to care about too many things i honestly had it easy latency and bandwidth consideration only happened for when i was working on working on low level engine tasks or maybe a rendering task or network task but not necessarily when working on tools or distributing builds identity and access management i wasn't involved in that that's for security experts right data coherency and consistency i only thought of that when i was thinking about cache coherency but not really at scale service deployment i got a fair share of that mostly outside of game but i know that all you guys working on game as a service are taking the the road to learning that we have to roll new versions out but we have to roll them back to we have to have processes for that and embracing microservices um it's interesting in terms of deployment uh stability i have availability but with that comes debugging distributed applications that's something new for me monitoring those applications and encryption as well isn't something i used to think about as a game programmer but as a cloud programmer i do need to think about that so let's dig into our first challenge the data game pipelines generate and treat a lot of data some engines are actually pretty bad at identifying their inputs and their outputs so the first challenge is well let's move all of it to the cloud while moving data into the cloud can take time uh you probably don't want to upload everything and moving them out of the cloud can be costly so on top of that if you add caches or on-demand requests to your data then maybe the cloud will add latency to that so it's all a few steps of problems that we're gonna we're gonna go through but the resolution is fairly easy basically when you design a system plan for data consistency and currency basically what are your users seeing at time t and how is the how is the data set current at every step in in your pipeline avoid designing systems that continuously require to push tons of data outside of cloud basically push only data and embrace laziness embrace on demand access the good thing about that is that's something we already do in games uh usually on game engines we try to pre-fetch the latency critical bits we tend to do clever paging and caching of the important data and we only touch what we must so let's apply that to data in the cloud it works fairly well the challenges number two is once you get your data in well you're gonna run executables over there but if you look at the lens at the landscape of cloud computing it drastically changed from a few years ago back in the days there were only vms and you would set up your linux machine and then you would do whatever you need to do to run your executables nowadays you have vms you have container engines such as docker with its orchestrator kubernetes you have fully managed tags like app engine flex and you can choose between a variety of options being sultenant on your machine so no one else can can run workloads on your machines or multi-tenant nodes where basically you share those resources with other companies you might have to think about networks cloud networks so really the resolution about that is to think about your workloads not all of them are created equal so what is the best stack for each workload if you're building a web page maybe you don't need to run that on one vm maybe you can use something that offload some of the infrastructure from your shoulders so you need to define the flexibility you need and the customization level you need from your stack with this comes how much time you're willing to spend on that technology and again we should take the road that lets us focus on our product we're making games not infrastructure challenges number three is i'm running a service in the cloud i put some data over there i solved my last two changes so now i need to present that to some users well that's easy right we're in the cloud so let's build a web ui well not so quick i often hear let's build a web tool for everything which to me sounds like let's throw away what we have and redo everything well don't do this never throw away everything not everything web isn't easy it's definitely not easier than your common application don't underestimate web it's a deep tech stack that you have to learn so you have to ask about web what does it bring and what added value does it bring to your users only use web when and where it makes sense for the rest use apis and endpoints that allows your original applications to do remote procedure course into your service cloud isn't web web uses cloud but not the other way around if you're interested in apis look into grpc cloud endpoints and kubernetes services and ingress it's really interesting it is a good step forward to you to be utilizing cloud compute without having to redo all the user interface change four is i have my service running everything's fine right well picture for a second your bug database picture that without the ability to inspect logs get call stacks reproduce the bag locally or even know that there is a bug happening well that's scary right software and hardwire fail all the time so the next step is because you're running workloads remotely you need to have signals about is it healthy is it working so you're gonna have to monitor your applications that's your second step once you get your workloads in second step is to start monitoring you gather metrics you define alets and you define playbooks fairly early on for when those alerts will be triggered it is important that the person who operates your service knows what to do when things fail and they will so for that i would say you can look into cloud monitoring but if you're more on the open source side of things and using kubernetes you might be looking at primitives and graphana next is security and access control that's the part where it starts to get interesting if you expose something to the internet well guess what somebody is going to try to gain privilege access somebody is going to try to exploit that or at least toy with it you don't want to have any one toy with your running services anything you expose to the public internet grows an attack surface and you should treat that carefully so as much as you can use private ips don't expose things to the pr to the public internet to do that you can use nat and gateways vpns if you are using clouds and virtual private clouds you can probably pair them together to avoid exposing anything to unwanted users protect your traffic with firewalls ideally use distributed firewalls that can protect at the vm level all your endpoints should do authentication and authorization don't trust that because someone gain access to one endpoint then it has access to everything else each endpoint should do its own authentication and authorization the last two points are about privileged workloads now if you're in the kubernetes world most of the examples you'll see are actually running your containers as root so with privileges you should try to avoid that look into pod security policies that can help you define that at a cluster level and make sure that your pods are actually secured finally encrypt what you can it is important not to trust that nobody is sniffing information out there most likely there is somebody trying to actually gain access to to what you have so take sector security uh with uh treat that as as an important matter and when i joined google i want to share that i discovered the beyond cop model i engage everyone to look into this if you're interested in security it's basically building a zero trust network it is very interesting challenge number six happens when you are deploying new application versions and new features okay we have our workflows our data everything secured thumbs up we're running in the cloud we need to deploy the next version seriously what could go wrong each time i submit something that's that's what i ask myself so because everything can go wrong when you deploy a new version there's a few things you should put in place to avoid things to be wrong and too bad have too much impact first thing first embrace infrastructure as code you don't want to be clicking on uis to set up your infrastructure it is very error prone and it is not reproducible so embrace automated processes and that goes to your rollout and rollback processes those should be automated you should be checking in configurations and a tool should pick up that and actuate your clusters define your signals for healthy roll out so that if that signal is bad the rollback can be automated if possible or at least you could be alerted that something bad is actually happening to your users never roll out versions to the entire population at the time i used to do that when i was deploying the editor build every morning to everyone sometimes it resulted in everyone not being able to work that's probably bad so even though you have testers probably in place to catch this early on an actual way of working in the cloud is to roll out progressively to your users a new version finally i want to talk about dynamic feature switches so we can roll out new executables we can roll out new data but that takes time and because it takes time it is an involved process what if we could turn an on and off features dynamically through configs that's very easy to build right as engineers it's a matter of if statement this is a very good way to enable and disable features you should if you can embrace that but if you do embrace that make sure you can trace both the versions and which features were actually enabled at a time in your logs or in your crash dumps this is very important when you will be debugging a specific version of a crash to understand which version and which combination of flag where we're on at a certain point so make sure you can trace that challenge number seven and the last one for today is debugging everything's running smoothly i have monitoring in place i can know what features are in and what versions are running but i soon realized that attaching a debugger to a production environment is complex if it is a load often it's not um if you embrace micro services what is gonna happen is you're gonna have states that are distributed across many processes one of them might fail but when you're gonna look at the crash or the log of that specific application you might not have the full story about what was happening so you have kind of distributed states so it is important that you log everything make sure you you have a lot of information it is better to have more information and not use it than not to have that in information available when you need it try to correlate your logs based on application ids maybe a remote procedure call ids or session ids if it's a session based workload that you're running try to embrace unit testing i know this has been a long rant about unit testing in the past decade in games but it is better to catch bugs sooner rather than later fuzzing your apis can help as well and basically do load testing you're running in the cloud you want to service a given population make sure you can another thing is don't be shy create custom tooling that could help you catch bug sooner or fix them faster so analyze what takes times in your workflows and create custom tooling to be more secure and more confident about how you fix bugs in production when all things fail embrace forensic debugging make sure you have enough data in your logs in your databases in your crash dumps so that you might not need to re-throw a bug it is actually better to be able to write a unit test and fix that with confidence than to try a repro that only happens uh one time out of a thousand in production servers only so in conclusion i hope i didn't scare you there are a lot of changes coming your way with the adoption of cloud but we've seen that the game industry as a whole is embracing this new opportunity whether that's through game servers or through new pipelines and engines it's coming our way we are gaining new flexibility in resources we are gaining new ways of working we are getting there but with that those opportunities are coming new changes uh whether that's latency and bandwidths a lot of platforms we have to find the best from and choose the best for our workflows running lc services and monitoring them managing deployments and enabling features and checking that stability is always good so you might know the the saying about you can teach an old dog new tricks i think that that's false that's we all dogs in the game industry should learn new things from the cloud industry this is definitely something that i'm super excited about and i think it's a very exciting time to be in this industry so with that i thank you thanks for your time i hope you're enjoying mix and i'm available for q a