Performance profiling solutions on Stadia

Key Takeaways

[Music] hello and welcome to this session dedicated to study as profiling tools i'm pirik software engineer at google and joining me today is michael antila from the stadia graphics tools team today we would like to present an overview of stadius profiling tools i will first present orbit our cpu and system profiler before handing it over to michael to present dive our new gpu analysis tool let's start by looking at the cpu side of things the very first thing i want to mention is that if you've installed the study sdk then orbit is ready to use it looks like this orbit is a feature-packed native cpu and system profiler that connects to a study instance and displays live profiling events as a user interacts with the game we'll cover most of orbit's feature in this presentation but before that i'd like to get back to basics and outline the core idea behind the tool first of all why do we profile well it's no secret that performance is paramount in games and not only that but good performance on average is not enough a single frame taking more time than it should will break player immersion what we're aiming for is for consistent performance also profiling allows us to make sure that we're using the available resources to their full potential being able to visualize time during the frame where the cpu is idle can help us guide optimization efforts by offloading work to other cores for example and finally profiling helps gain a deeper understanding of our own code it allows us to build an accurate mental model of our complex multi-threaded application to me a very good way to build this mental model is by visualizing function calls we can see that the code on the left can be represented by the diagram on the right in orbit we call this diagram a flame chart on a flame chart the x-axis is time sequential function calls will appear from left to right the width of a rectangle corresponds to the time it took for the function to execute the y-axis corresponds to the function call hierarchy where nested function calls will appear from top to bottom visualizing multiple threads is also very intuitive we simply have multiple flame charts on the same timeline this can be very useful to visualize interaction between threads to again help us build our mental model of what's going on in our frames now the question is how can we generate those flame charts well the easiest way would be to modify code however this would mean that to instrument a new function we would have to shut down our game change the code recompile relink redeploy relaunch this can easily take 15 minutes on a big project also performance problems in games can be highly contextual you might need to reproduce a long sequence of events to trigger a performance bottleneck adding to this iteration time another problem here is that you might be interested in profiling modules for which you don't have source code so how can we generate these flame charts without relying on code annotations orbit's answer to this question is dynamic instrumentation it is the number one feature of orbit it allows us to generate flame charts without having to modify a single line of code there's no integration time virtually no iteration time and you don't even have to relaunch your game in orbit you can select functions to dynamically instrument or hook and on the next capture the timeline will be populated with all occurrences of the hooked function calls this information will be presented as flame charts in the capture view as we can see on the left note that this is exact information this is not statistical data like what you would get from sampling under the hood orbit is actually intercepting function calls and is getting a timestamp on the entry and exit points of instrumented functions in addition to dynamic instrumentation orbit does offer a powerful sampling profiler sampling by the way is the process of periodically getting a call stack for all running threads sampling is always on in orbit as opposed to dynamic instrumentation which requires user input in the diagram we see that every sample represented by a yellow arrow produces a call stack in orbit we can visualize this call stack data in different ways first we present every single individual sample on the timeline as a vertical white line this is what we see on the sampling bars on the left on the right side you will find three different ways of looking at the same sampling data we have the flat top down and bottom up views the flat view is a flat list of functions which can be ordered by how often they were present anywhere in the call stack or also by how often they were actually executing at the time of sample note that it is possible to generate reports for a selection of samples by interacting with the sample bar the top down view presents sampling data as an intuitive call graph so basically expanding nodes will let you traverse the call hierarchy from parent to child the bottom up view on the other hand allows you to see which functions are being executed the most in other words the function in which your program counter was at sample time expanding the nodes here will let you traverse the call hierarchy in the opposite direction so from child to parent so dynamic instrumentation on its own is great and so is sampling when you combine both you get something that's worth more than the sum of its parts it unlocks a workflow that makes iteration incredibly quick and this combination of sampling and dynamic instrumentation is orbit's core value proposition and a good way to understand this value proposition is by putting ourselves in the shoes of someone who knows nothing about the binary they want to profile how would we go about populating a detailed per thread breakdown of important function calls well we start by taking a capture without any dynamic instrumentation we then analyze the sampling report to identify functions that would be interesting to instrument we would then hook those functions and take a new capture very quickly we'll populate flame charts for the different threads of our application we could iterate on this loop a few times to make sure that we've identified a good set of functions but in the end we'll have generated a detailed capture in no time and again without having touched a single line of code once we're happy with the set of functions that we've identified for our profiling session we can save them in a preset so that we don't have to manually hook those functions again on every capture this will reduce iteration time even further you can then simply launch orbit selected preset and capture of course if you run into a new performance problem along the way you can always add functions to the preset so a quick note on debugging symbols in order to profile we need them this is orbit's only requirement they can live either on your instance or on your local machine if they're on your instance orbit will transfer them to its simple cache on your workstation so debug symbols are used by dynamic instrumentation to know where functions are in memory they're used by sampling to do the mapping between an instruction and a symbol and they're also used by our source code and disassembly views for line information so now that we've covered the core functionality of orbit let's have a quick look at some of the other features it has to offer in the capture view the first track that you will encounter is the scheduler track at a glance you get an idea of your core occupancy you can also easily see what core your threads were running on and visualize context switches related to scheduling information we show thread states right above the sampling bar in the thread state bar a thread that is running will be shown in green a thread that is runnable but not scheduled would be shown in blue a thread that is sleeping would be shown in gray you can hover over the different sections of the thread bar to get a description of the thread state to keep an eye on memory orbit has dedicated tracks in the capture view to trace memory usage and page fault information for more info on this you can hover on those tracks to have useful tooltips i would now like to talk a little bit about manual instrumentation so we've seen that dynamic instrumentation is the number one feature of orbit but manual instrumentation can still be useful for example you might be interested in profiling sections of functions or profiling asynchronous operations that would start in a function and in another one or you could be interested in visualizing the state of variables directly in the orbit timeline and for that we provide a manual instrumentation api we took great care in making sure that integrating manual instrumentation in your code base is as easy as possible there is no library to link with all you have to do is include a single header file or bit.h and use our simple api in your code it's worth noting that our manual instrumentation api made its way to the unity code base the scopes that you would see in the unity profiler also appear in orbit respecting unity's color scheme similarly you can easily add orbit markers to unreal's existing profiling macros in order to visualize unreal's instrumented functions in orbit another interesting feature is the disassembly view on which results from sampling are overlaid this is useful to visualize hot instructions similarly the source code view allows to map samples to lines of code coming back to the capture view we have function iterators that allow you to define a frame that span consecutive function calls in this case we've used vkq present as our frame delimiter we can then easily navigate from frame to frame in the capture view so on the subject of vkq present while orbit is primarily a cpu profiler we do expose some gpu driver events we have a dedicated gpu tracks where you can see when a job was enqueued to the driver when it was in queue to the gpu hardware and when they were actually executed on gpu you can also use our vulkan tracing feature to visualize vulcan debug labels directly in orbit alongside all the other profiling information that being said for a more in-depth gpu analysis we do have a dedicated gpu profiler dive which michael will now present thanks pierreck a gpu profiler is critical for developing or porting resource intensive games on any platform and stadia is no exception when developing dive our goal was to create a profiler that would give you more information than existing gpu profilers both at the low level and at the high level we recognize that there are two groups of game developers in our target audience those who are optimizing the underlying graphics engine and those who don't have any control over what the graphics engine is doing i want to show that dive can be a very useful tool to both of these groups dive was designed to be used full screen on a large monitor in order to provide the best experience possible so for these slides i will only be able to give you small glimpses into the big picture with that in mind let's begin dive is a single frame profiler you simply start your game up on stadia launch dive on your developer machine and hit the capture button dive will capture performance information for a single frame and then transfer that data back to your workstation for detailed analysis dive has a typical timeline view of everything that happened during the frame and an occupancy chart which gives you immediate insight into how busy the gpu was at each point during the rendering of the frame existing profilers have occupancy charts but an overall occupancy chart can only give you so much information what does 50 occupancy actually mean are half the compute units idle or are all compute units being partially utilized these two situations require very different optimization methods for game developers who need to know exactly how their graphics engine is using the hardware existing profilers fall short of giving them enough information to answer questions like this dive allows you to drill down deep into the gpu into each shader engine each compute unit and each cmd at each level you can see exactly which resources were in use and how occupied they were we also have instruction tracing so dive can show you when each individual shader instruction was executed and exactly what that instruction was for graphics engine developers this is the level of detail you sometimes need to track down an issue dive shows you the state of the hardware context in a timeline view and can automatically highlight events that were stalled due to context rules this is just one of the ways that we try to bring your attention to potential performance issues as quickly as possible at a higher level dive also captures which vulcan calls were made during one frame and displays them in an event list on the side one feature that says dive apart is that we capture all of the vulcan calls not just the ones that result in gpu work this extra context is critical to understand what is happening previously developers had to perform an additional capture of a frame in render dock just to understand the context of what they were seeing in their profiler this feature has been a huge time saver debug labels are captured and displayed in both the event panel and the timeline panel for easy cross referencing debug labels vulcan calls and gpu events are organized into a collapsible hierarchy for easy navigation and yes we have a search bar so you can find the exact function you're looking for every parameter to every vulcan call is captured so there is no guesswork about what is happening although dive is not a debugger this feature can be useful by itself for figuring out where things are going wrong in a graphics engine we capture as much information as we can you can view the contents of every command buffer that was sent to the gpu this has proven useful when trying to get to the bottom of some really interesting timing issues including allowing you to see which copy buffer calls are going to dma and which are being dispatched directly to the gpu in addition you can view the gpu state at any point in time during the frame this can be useful to verify assumptions about exactly what state the gpu is in dive can also capture performance counters from the gpu sometimes just staring at a timeline of events isn't enough to give you insight into where the bottlenecks are performance counters can help identify draw calls that might be problematic for example sorting by the amount of time spent in the vertex shader will instantly take you to the call that is doing the most vertex shader work this can be a very quick way to determine if polygon counts need to be reduced for some models in order to hit performance targets for developers who don't have access to the graphics engine code this may be a good opportunity to go back to the art department and see if they can simplify assets to improve performance in this example looking at the counters for this call dive can let you know that only 43 of the vertices actually made it to the final output maybe this is an area where some culling can happen on the cpu or in a compute shader to avoid having the vertex shader waste time on all those unused vertices graphics profiling can be a challenge and we wanted to make dive as helpful as possible we have a context-sensitive help system which gives you useful explanations and advice as you hover your mouse cursor over various elements in the ui in this case hovering over a barrier reveals what that barrier is doing and how it could potentially be optimized by doing things a different way we also have a problems tab which can identify certain common performance issues that we see frequently in games in this case dive has noticed sets of barriers with no draw calls in between them which means they could theoretically be merged into a single barrier to reduce overhead hovering over the mergeable barrier set brings up context-sensitive help text that gives you an estimate of how much time you could potentially save by merging those barriers in this case 0.11 milliseconds which could be significant savings if you are close to the edge our ultimate goal is to automatically find as many performance issues as we can on your behalf to help save you time and help new developers learn about what factors can reduce rendering speed we plan to add more static analysis features like this in the near future in addition to profiling dive provides a user-friendly interface for analyzing gpu crash dumps dive integrates with the graphics flight recorder to provide as much information as possible about gpu crashes graphics flight recorder or gfr for short is another tool available in the stadia sdk that can log the state of the gpu as it was just before it experienced a hang or a crash using data from gfr dive is able to quickly highlight which falcon calls completed and which did not in this gpu hang case dive was able to instantly highlight that a copy image operation did not complete for some reason drilling down to a lower level dive can identify which shaders were in play when the gpu crashed and identify which individual instructions were in flight across the gpu at the time of the crash hopefully helping you narrow down the exact cause of the crash as quickly as possible we hope that by giving developers access to as much information as possible and organizing it in a way that makes it easy to analyze we can make profiling and optimizing your game on stadia an enjoyable experience and we are looking forward to hearing your feedback on these tools as you use them thank you for watching there is an open source version of orbit available on github and both orbit and dive are included in the stadia sdk thank you for watching and please try our tools out [Music]