Inside TensorFlow: TF Debugging

Key Takeaways

well don't you do this presentations about how to debug tensorflow programs we're going to focus specifically on tf2 because it's you have to disable our release and it will have it will have long-term support going forward but there are also places we're going to mention on tf1 and when we do we'll make that clear so you know which version of intensive one we're talking about so first of all I want to define a scope of debugging and the reason why you can do this is because the word developers have overloaded scrummy in machine learning different people use it to mention or to refer to different things sometimes you confusing ways so in the scope of this talk debugger will do specific things only that I have that many have to do with the correctness of your pencil fill program like mathematical implementation books for example when you Laye are implementing a new layer type or new loss function you may run into D type issues or shape issues or just straight bugs in the math and the techniques will cover will also be useful for debugging the pre-processing and post-processing parts of the attentional program and one big focus will be the debugging of the issues like there and infinity your models which have been very frequently during your TF model training Mehmet will talk about a specific tool concentration which is very useful for catching the root cause of men's infinities on GPUs and other directors and we're not going to talk about how to debug on the magazine called kernels themselves or magazine hardware because it's more specific to the hardware or the kernel intentionally using Heiberg the methods we'll cover will be useful for you to debug in bottles that are affected by those kernel or Hardware bugs at least it will be useful for you to like narrow down the cause of the model back to the level of acronyms for hardware and that Dom tools and techniques will cover will also be useful in case where you'll want to just peeking into your model to understand the army meter what's going on and so one example would be answering question like why is my model making a wrong prediction I spit in input examples so you will be able to like peeking through the model and look how the layer activation is intermediates tensors and answer that question so one use case that's kind of relevant to that we are courting from one version of the library to another or from one library to another like from personal to TF lights off from intensiva to PHAs or from ten seconds apart watch or on from package to hazard log on you will often see on divergence between the two implementations of the same model and you want to quickly identify the root cause of the villages and the tools and techniques of all color will also be useful for that purpose what we're not going to cover are the debugging cases like debugging the model performance and looking at the accuracy of models after training like on model combination and analysis we're not going to cover how to debug feminist issues with models of those are also important kinds of intensive period debugging but those are outside the scope of this talk there are great tools for those thanks on some - more intensive wards and what if - oh and fairness to infinity so forth but I'm not going to talk about this any questions so far okay so here's a brief outline of this presentation we're going to talk about how to debug test O'Reilly we're going to talk about how to look at the placement of apps on different devices like CPUs and GPUs it's a very commonly asked question we're going to look at how to debug the structures of graphs intensive flow - including the graph frontier functions and graphs that are optimized for the long time and then in Section four we're going to cover in a special topic of step debugging which is to use a tiny taste ethical code line by line and then in the fifth section we're going to learn from low-level api to evaluate the api and a specific article focused on is the other county of chemists because TF harris is the official hannibal api in tf2 and also because I'm not that familiar with other high-level events like Sun and so forth and in Section six willing to talk about the division of numerical issues like Nandini and finally I'm going to present it to all health tensorflow debugger including the existing v1 features and the litchi features that we're currently working on so first let's take a look at how to give intensive values so here's a very simple example so it's very straightforward not separating urine functions with GI function and decorator and so everything is excluding healer in tf2 and there you can simply use the print statements to look how the values of tension spiking this simple example here so why is an Egret answer if you do print and will see the value in the step out printout so it's very similar to print the value of the vampire ease with a caveat that's if the tensor that's on the GPU then printing it will involve copying from the gph new your host which may be a little costly and often times when the size of the tentative you probably don't want to look how the entire tensor because they are going to be millions in elements what sometimes we want is to reduce operations on the tensor and then look at some numerical summaries of attention like what's the minimum value what's the maximum value and what's mean it's also a useful technique that's fully supported in either mode so eager tensors string and repair methods use vampire under the hood which means that you can use the set prints option function from them how to control the details of how the values of tensors are painted for instance if you use program the precision few arguments you can adjust basically how many decimal points are printed in the flows type tensors you can also adjust like the threshold element accounts beyond which ellipses will be used in the printing which is useful for israel you want to look at the values of huge tensions like thousands of enemies of course storming is not always as straightforward as this the program is often not excluding purely evilly and sometimes you have team functions and sometimes your function is decorated by tensorflow itself in converting into the graph so there if you use the print statement then the result of the printing may not be what you expect so here the user intent to look at the value of n at each iteration of the while loop so the user puts a print statement here naively and the result is only one printed line even though the while loop is executed multiple times and the contents of the printed text is also kind of confusing to a naive user and the reason is that when ta functioning is used the code here is transformed into a TF graph and the print statement is executed during that function to transformation and the contents you see here is actually a node in the graph instead of the valuable of the tensor at one time so the current approach here is to use TF print so TF print will modify the graph will actually add a couple of notes to the back so you can look at the value of the end inside of the TF function so here on at the bottom here you can see the value of n at each iteration of the is printed so this question is for so here's a homework problem for you so the examples I've shown so far are four simple tenses like flows 32 tensor or an integer tensor what if the TF print statement is used on the rack tensor or sparse pieces so those are the major icon composite tensor times T as it looks so we can try that on by the way I inserted a link to the collab notebook for all the color examples in this presentation so you can kind of slides and if you are if you want to play with the code examples you can use that notebook so here is a second home working on the code you can use the code to see what happens if you use the T of the prints now let's first answer okay so sometimes the user doesn't want to just print the valuable time so the user wants to programmatically extract the value of the tank so they can be used for like a programmatic debugging or downstream competition this constant period shows how you can pull out intermediate answers from a toy implementation of an aunty of dense there so the function originally returns only the final outputs of the density but maybe for some reason you want to know parent intermediate steps like the results of them as well or the results help that adding with a bias so what you can do here is you can actually append these two tensors to the return values of T n function and then you'll be able to access these intermediate values where you call them what slightly more complicated is if those tensors are inside control flow structures so for instance if you want to programmatically access the value and at every iteration of the loop you can't simply just add n to the return value here what you need to do here is to you see attention rate and then appended up defensively as each iteration of the loop and then you will be able to see the folk history of college educators it's starting complicated so on the attentive lot of efforts to I will present at the end of this talk I'm hopefully will make this super so having covered tensor values I'm going to talk about how to debug the placement of apps on different devices many CPUs and GPUs so it's a very frequently asked question becomes a user's long to make sure that's their heavy competition is computed on the GPU so again if the program is running purely agree then it's pretty straightforward you can just call an API call TFW downstairs lock device placement equal to and then when those operations are executed eagerly it will see like spring being printed to the step out for instance when the multiplication operation here is blend you will see on line that tells you that the mountain mall is running on the GPU and when the the print statement here it is running exactly running a príncipe to the rock on the city so here you can see clearly like on where the hogs are running whether its own CPU or GPU and if you hold if you have multiple GPUs which of you is running on so one thing you need to know here is that it's only going to have a lot of information when the app is placed for the first time if you have the same up in excluding multiple times unique it's not going to print it on multiple times so that's an intensive prevents a flood of information to your on step out so a more realistic scenario is where you have TF functions and graphs so on there says lovely device placement equal to true will still work it will see not only the placement for the big rats like in the green box here but it will also see the placement of the graphs in the purple box on the bottom view so but one caveat here is that you need to be awareness even though the eager lines are printed to store our country the graph logs are printed - viva la the implication of that is that if you are doing this you know on your notebook or Colet then you will not be able to see the bottom part something lock but there is actually a way in collab to capture the locks you can see of it's just i'm something you need to be aware so in the graph placements around lungs the text inside the parentheses are for the up type and the text outside of presences to the left of the parentheses are for the name of our so here are some important things to know about some stats luckily the best placement so it works for both eager operations and for graph operations but they're allowed to difference places as i mentioned and also be aware that the fact that an app is locked at graph construction time does not guarantee none the better off we'll be experience from runtime and that's because tensorflow has this built-in graph optimizations jack conte grappler and grappler may change the placement or it may pruned away from the ref or it may fuse the into a larger and so forth i'm going to talk about that you know come inspect and also be aware that I'm set Lock device planners country does not work for me for to be used so it's very useful for debugging views and GPUs country okay so I'm going to move on to a section about some debugging graph structures so here you have a TF function and then how do you look at the graph of the composition of those on TF function so the answer to that is to use the method call to get concrete function on that function object so get country function should because only after density function is called for the first time if you call that before the function is compiled then on that method will not even exist and when your call has been preached function we need to pass an argument so the argument can be the same argument as you pass where you're calling the function and the reason why you need to pass the argument is because on the same type of function can be compiled into different TFS depending on the de types and shapes of the hippo animus you can also pass ten suspects as the arguments and the return value of Gascony function here is an object that has a graph property the graph poverty is a TF graph on to see its structure you can call as graph that letter to the graph object and the return value is a text color for the graphics as shown on the right here so the text point here is basically repeated fields of notes it tells you which notes there are in the graph and how they are connected to each other so there are properties like the name and all in the attributes and so forth so if you're not familiar with the format you should spend some time then looking at some examples because it's very critical very important for a supplement but the important thing here is that for any realistic models and we list a few clear functions the size of the graph type is going to be too big it's not going to be friendly to like on reading a text and that's why our graphical tools that can support it will be important so you can start to test board the binary so even though you have even if you have an empty doctor you can still switch to the graphic dashboard by using the drop down menu and then inside the graphic - but you should be able to see a bunch of the countries file and you can use choose file to upload the contents of the PP text of graph and then ten supporting won't be able to show you the structure of the path that's true this is example here so some important properties to know about tensile force graph on dashboards - the information flow is generally from bottom to top so the inputs are usually on the bottom and at the top here seeing the outputs and also ten support perhaps visualizer work group notes by link scope by default - so that's the reason why you often see those like big colorful boxes you can double click each box to expand internet so it's quite handy for debugging large ones and it also handles function deff library car keys so function that libraries are basically nested craft so it's used frequently in tf2 like in control structures so like a tf2 while we log contains two function that is one for the condition of the while loop and the other for the burning of the lung those are also become color boxes that you can double click to expand eg if you're a Google internal ok you should be able to use a special import I'm from collab and that will enable you to look at a graph instead of collectibles book so I find that to be slightly handier then like looking at graph structures intensive myself because that means I don't have to switch back and forth between two different types of a browser so here is an example so as we mentioned before you have append variables to the returned list of the tail function to access intermediate tensors and in this graph being visualized by the tensile board you can see two extra and he notes that correspond to the added on return values and that's the house on tf2 currently uses identity notes to mark the return values of T other functions so here's a function here's a graph for a function that's slightly more complicated so it involves control flow structures including one of those tons of else conditions so these are the boxes without other functions that libraries actually work so you can see a box for the true branch of the if-else collision you can see another box for the false branch of the get bells condition and the box here in red is the condition of the while loop and so forth so if you are very careful and if you spend some time you can see the correspondence between these arcs in the graph and also in the operations in the Python code but that's in general how to do and it requires an army expertise to see the first moments between the bathrooms and the operators or factory functions so that's one cat we're on the TF debugger on tool that I will talk about tries to film so in TF debugger tool you will be able to the kind of graph structures and the source code comes it's a light side so it will be easier for you to establish the correspondence between the Python functions of Python operators and nodes at the back okay so what if the function is not executed on a single device but on this excluding on multiple devices or multiple hosts using a distribution strategy so before I talk about that I'm going to tell you about a useful API for blocking outs from virtual devices so first is if you want if you have only one physical GPU on your machine and you want to do some testing or some debugging on a distributing strategy that involves porting different GPUs then you can use the API set virtual device configuration to create for more yes for long college views and you can use API catalysts logical device to confirm that it's a very useful technique for testing and debugging physical functions does involve multiple devices so once we have said that for lock code abuse we can use a different strategy to basically on create variable on the floor GPUs and we can construct a function that's a world basically call gradient tape on that variable over under for GPS so the function here counts on this underscore F is the function that involves the replication and you can use the get concrete function method as before to look at some graph structure so you can upload the gravity PD text to pencil going to see structure and in a structure you can see four boxes and those four boxes correspond to the four GPUs so the technique here is useful for debugging graphs in there are strategies and other distributions okay so this side here shows you how to have them print to work in terms of the path so each time you can't you have to print inside a TI function it will append a pair of nose to your graph so the first node here will convention or tensor that you pretend to an integer stream and the second one will actually print that string issues that out or interlock or on whatever alpha stream the the treetop is configured so here's a for example for you I mean it's also available in the collab those book so you can play with learning and that's very interested so the question here is what happens if there is no return value from the function so I forgot to mention that's the reason why these prints of e to ops get executed at one time is because they are attached as control dependencies to the final housecoats identity note of the graph so these correspond to the dashed lines in the graph so the homework problem is about finding out how the print gets executed when the TA function it does not involve a mature manner so when you use T after Prince you need to be aware that if you inadvertently change how graph optimization works as when tightening so in the cost limits on the left here were computing the harmonic mean of a tensor however there's a line in the code which instructs it up but the appetizer that's obviously not this is not using any down student commutation now when the clear function is expert at one time grappler has launched into his job he's going to prune out Catalina so the meal will not actually get executed at one time however if you use the after print you will change the optimization autonomy and you're basically going to attach the upper tension of the way up to the string formats and the prints of each other's eye as uniform but important thing to note here that if you use the best concrete function method to develop the graph structure you will always see the middle and that's because at the Python level intensive low or graphical faithfully only converts the petal function into a graph it's not trying to do any optimization instead it will hand the graph to graduate for downstream optimization so the pressure here is called can we debug the optimized graph that are generated by a grammar so that leads us to the next section so in order to see the grappler output graph you need to use on basil build so when you call this you need to specify an environment I will count TF dump graphic effects and points to any directory you have write access to and then you have to specify the flag convey module so that tells the meta optimizer which is a part of grappler to be verbose and dump information to the folder and after the program runs you will see a bunch of files in the folder so those are the outputs from each pass of graduates oh brother performance graph optimization in steps kind of like a compiler so the final outcome which is usually contact or meta optimizer something is usually the part of interest so it will tell you the structure of the graph that gets executed at one time so using a technique here you will be able to compare the runtime the two constants that we have seen before so either constants depends on the left and also the graph on the left you see that the mean up is not present because it's pruned away by trapper over in the console if it's on the right here and also graphically right here you can see that the mean up is present and feasible into that she offs that correspond to dferent so as you can see that the process here has come to it and complicated so TF debugger will try to present both the Python graph and the runtime graph to you so you don't have to do any unbound bills or any like on special flex or immigrants vitals okay so now let's talk about the interesting topic of step keep I mean so by step debugging I mean using a Python IDE or PDB to step over lines of the source code one by one some people prefer that over our principle deeply so the useful API here if you want to do start debugging is the TF dolphin Victorian experiments on run functions degree so if you call that function with the unemployment route then you're basically calling you're basically telling the TA functions you do not compile function to do graphs and all the code here will run eagerly and you will be able to use either print or you can use step debugging or if you use breakpoints in your favorite IDE but one important caveat you want to keep in mind is that it's worse for our places except TF data data because tilde that are works in a special way it always converts Python functions into graphs before ends them I'm going to show an example for that in an upcoming slide this record shows an example of using the s code to instead debugging on a TM function actually you called experimental run functions in a little tree however if you don't call that some function I mean if you don't call it experimental run function is ready or if you're caught on its false then it's not a good idea to use that give up your on th function and for summary if you add a breakpoint it's not even going to hit that break point and in other energies you will hit that spring points but the stepping pattern after that's a great point it will be very confusing and the reason for that has to do with the internal details of how works refer you to the presentation made by 10 long event last year I think it's publicly available so understanding that it will probably not be too hard for you to understand why the strange behavior is having it having so the strange behavior is that you're inserting print statements in both branches of the palace condition and you see that when the function is called both branches get executed yes so the citations using an example in which experiments of run functions degree does not work on a map function that you pass to tease out data set so even if you comments out that you have function decorator for the two multi-college function it's still going to be converted into a graph and then run in a graph fashion instead of running it early so in order to debug intermediate tensors inside a function you must use thi friends if we do prints you're only going to place the symbolic tensors in graph but just kind of load debugger will also make it easy for you to debug the values inside a function has two TF okay so so far would have been talking about how to debug low level constructs on both kinds of low including artists and tensors in the class but many users and also use hallabong against like um here Karras and then they also want to peek into their own so you don't fall in space I'm going to talk about some tools and techniques available for people in karin-sama's so one other I frequently asked question is how do I that's the intermediate their outputs I mean the intermediate in their activation frontier of periscope so one way to do it is to construct a second model which is the deep background in example here the second model has the same inputs as the original model but the outputs will be the original model output plus the options from the layers university and then when you called debug model that predicts or simply called debug model as a function you'll be able to see not only in front of the Charis model but also the intermediate so this approach is useful when two is useful to look at the final purpose of each there if you want to look at the intermediate answers inside each where it's not that's become in school you have to use it only TF print methods or the other technique specially in earlier parts of the presentation and TF debugger will also make it easier for you to mcadams layer internal issues for me there on even tensors so one other useful thing to know when you're debugging at TF cards model is to use the ten support come back so the pencil or come back which is under the tiempo para stuff public's namespace is a call back you can you pass this URL fetch what it will do is it's going to love last functions and metrics to block there when the bomb is free but for debugging purposes it will also lock the graph of the model to the long term so you can just open the graph dashboard of principal and look at the graph structure so there you see that the layers of the model are organized in those boxes that you can double click to expand and thats thanks to the work done by the authors of TF carrots who have been very carefully specifying the correct name scopes for each year but the other important and useful thing to know is that the tensors are marked as those arrows that's connected the layers and if you look carefully you can see those very small fonts those small fonts are the shapes of the tasers to the extent know where to grab construction time the list is from the drop out there he refers to dimension x websites on question map x fight so the question is the undetermined vaccination okay so having covered on high level API debugging let's move on to the next section which is about how to debug on Nancy and infinity senior so that's a very frequently occurring debugging test and as it long and it probably accounts for about half of the questions that we get asked so before talking about the tools I want you to show you some common causes for an ends and the infinities intensive level models so they can be caused by a lack of data clipping like when you have a division operation in your own pencil program like some sort of like organization if you forget to add a epsilon or a very small positive value to your denominator it's like each Alliance you at one time especially in the face of the variability of input stream they don't that also applies to the macula operation and sometimes the density and evidence in your pencil mark can be caused by bouncing up criminals on themselves or even Hardware so for instance we have seen about recently that it involves national Colonel on TPU you start putting infinities and then even when the inputs are totally invalid and sometimes the man's infinities can be also caused by exploiting gradients especially when your knees too high so in that case - oh the famous bean just keep coming and decrease the rate and then as Mehmet will tell us about sometimes the lens and fins can also be caused by traumatic training examples so debugging that which calls on Tina and the dignity is different from the Prince of debugging at the grass structure depending the cocktail party or your personal presentation and that's because time to find the root cause of land abilities you don't know where to look because that's exactly what you are trying to find out you could like insert your in statements are true every single tensor in your model but it's not going to work for online illness which can include up to like tens of thousands of tensors so that's why we need specialized tools to help you develop those causes and ends in the danza invitees so I'm going to protect two tools here the first one is a new API it's called TF debugging enabled check and Rex so it's a relatively new API it just came into existence ETF 2.1 which was released about a month ago so what API does here is that you can simply add one line of code to your TF program and when the TF program runs like winner model treats its going to check every floating type tensor a Orrantia program including the eagerly computed tensors contentious besides graphs and TF functions and as soon as any floating type tensor contains an Enzoani to be in their output then the program will around with a helpful error message as the one shown on the right here so the error message here contains a bunch of useful information for including the name of the app the runtime we typed into the shape of the leader tensor as well as our spectryx so we know that the stack trace from offensive low error messages are usually very rebels in how to understand and the API here tries to infer want to try to guess which the frames of the stack trace across come to the users own program and that's those frames where the error so hopefully it will be easier for you to find it important to frames in this actress and the API here is also general in a sense that it works for both forward pass that broadcasts it works for low level and high level equally Paris also works if you are stuck with a tf1 API and it should work on CPU and GPU and CPU so what question you might want to ask is what's the performance overhead of this and it's an important question because to find the root cause of lens and the overhead needs to be as low as possible sometimes the Nan's a name for these don't happen until like a few hours or even a few days into training so thanks to the work of an intern Anthony the overhead of this API is low so we have benchmarks on the API on a bunch of models so here's an example from the transformer v2 model where's training on the CPU if you enable Gus check about 30% overhead if you own if you're the if the mom is training on TV then overhead its lucky hiring it's about 75% but it's not that high so it may be even a good idea for you to turn this API on in your tests for 30 checks this API here is useful but it's also limited in the sense that it only tells you what happens when the LAN or infinity happens it tells you about the op but it has no information about the moments or the history of the exclusion meeting opportunity to dance moments okay so what tense ability whether is can be thought of as basically a combined tool that will help you achieve almost all the debugging tasks that we have measured in earlier process of the presentation including look at looking at intensive values on the placement of some devices graph structure also step debugging and numerical issues like Nana please so far we haven't put a lot of thought into high-level DPS product errors but it's on our radar so there are two different versions of MTF debugger b1 and b2 solely one was on a part of the tip 1 so it's centered around the ODF does such an API so it's basically a set of wrappers for your sessions so it's still available in intensive load if you are still using tf1 api so it might be useful to you so there are two different wrappers the camera interface wrapper and on pencil or record when you wrap the session objects you don't have to make any other changes to your own testicle codes when sessions are run runs is going to present you with debugging information if you use the canal interface wrapper then such a diagram will basically drop into an interactive on terminal based programming or on criminal and these screenshots showing you that the the Google interface will show you the list of tensors that are executed you can click those tensor names to look at the details of the tenseness like the placement the values of the tensions and so on they will also show you on source code and annotate each line in the source code with the apps that have created at that line so currently we're working on v2 of the weather the reason why we want to invest in this time obviously first Google should bring the tool up to speed with the current API which has no T after session but it involves you execution MTA function and also in earlier parts of the presentation you have seen that print debugging and TF prints are useful for a lot of debugging cases but it's not useful in all cases especially when you want to debug on some code deep inside the technical code base itself so we also want to incorporate some lessons we learned from be one of the twelve of course if we want a tool to be generally enough to work on all hardware types if you have given only one because it predicts on GPU in consequence um does not work for TV it only works for civilian GPUs like here even the v2 will work for all the major hardware types city was TVs and TV news and I'm certainly we want to overhead to be as low as possible we learned that there are some improvements that we can make make sure the UX of the front end so PFD biography - in a nutshell will involve this process so the user has a TF program that he or she wants to debug then they can just insert one line call into their TF function and specify a long term so the Lochner can be the same locker as your intensive board long term and then if you're a supporter has started and you can switch to the debugger eternity - - in consequence you do look at debugging information so the front end work is currently underway so I've been reaching out to various people like people at the testicle team and outside and soluti next Google to get their feedback if you're interested in dog food enos or telling us about your specific debugging uses please reach out to me and I would be more than happy to work with you to make sure that let me you tool will be useful for your problem so here are some UI marks that's on the US researchers help us make so it's going to be the look on the muon debugger v2 plugging in pencil board is going to show you the expression history from the top it's going to showing you on both eager execution of pops and the TI functions and you can zoom into TF functions to look at the graph structure and the list of tensors but a computer inside the tear function and the top left left section will highlight important events like the generation of names and infinities and repeated function compounds which might hurt your own performance and more importantly on the bottom section you will be able to associate your graph hops with your source code or associate's degree institution been successful this will make it easier for you to find the bug me find the way back from your bug into that source code and just speed up your about fixing prices and the finally some advice so the authors of testable have has done a lot of work recently to improve the error messages so next time you get an error message intensive low be patient and to read through the error message especially intersections labeled as a user code it may contain some surprisingly useful information for you to debug their problems and lastly on some machine learning got announced machine learning but they're just general programming backs so here the puzzle 3-2 on Chuang it's a small problem so I'm here the user is trying to go to this default or the features and for the labels and the user feeds them into functions you can set up the data set and the data sets is that into the fit call to the trainer model but for some reason the model training is not very good the oppressor is much worse than expected and what's the reason for that so it's a puzzle if you're interested in the answer reach out to me and I will be happy to tell you this but the point is that how symptoms are just general programming that's pressing thank you very much for attention [Music]