Inside TensorFlow: AutoGraph

Key Takeaways

thank you all for coming my name is ann muldovan and today i will talk some i will talk about some of the internals and functionality of autograph now this is definitely not an introductory talk and if you would like to learn more about the background or the motivation behind autograph here are a few resources that I believe can help the talk will be otherwise fairly fast-paced quite dense I'm hoping we'll be able to get all of it in time but if not I'm hoping the slides will be able to serve as a good reference should you decide to come back and look at anything more closely I should caution though that I am oversimplifying a lot of things for the sake of brevity and time but the essential things are in there the talk will be structured in roughly in three parts first I'll talk some about some of the more relevant implementation details which are useful to understanding some of autographs behavior then I'll describe the various ways in which you integrate within which you in can interact with and lastly I'll go through various use cases that highlight what works what doesn't work common pitfalls how to stay away from them and what are our plans to eventually address them so let's begin with the with the implementation from a systems perspective this is roughly what autograph looks like in broad strokes we have the following going from the bottom to the top we have an infrastructure for performing source code transformations with various helpers and on top of that we have individual transformations for instance there is a separate transformation that handles function calls another one handles break statements and yet another transformation handles if statements and these transformations are independent in composable many of these transformations then replace your code with calls to special autograph functions we call them overloads or operators for reads for the reason that they are very similar to pythons overloaded operators now of those overloads there are the most interesting ones the ones that specialized on creating tensor Oh ops and lastly there's a high-level API that glues them all together and this is typically what you usually interact with as a user one last note that I should make is that of all these pcs only the tensorflow specialized overloads and perhaps the high level API only these are specific to tensorflow everything else are is fairly generic and reusable and we hope to eventually have them in a separate library that can be used for other purposes as well alright so one of the fundamental pieces of autograph is of course the source code transformation bit so let's look at that a bit more closely source code transformation is essentially what makes autograph a transpiler it's unit of work is functions that is at runtime a function is being converted into a new function so let's look more closely about rapport closely at that process it is roughly loosely speaking a five step process the the first step is to obtain the source code of the function now the standard Python library makes that easy for us it provides the inspect module which is built-in and it lets us do that this also highlights one of the fundamental requirements of autograph in order to convert a function that function must expose its source code that's typically true for almost all functions in Python although there are certain exceptions normally you can test this on your function by calling the inspect gate source if expected source returns data then Auto cache to define with it the second step in this process is to parse the code into an AST and once more there is a standard Python API for this which is good we in fact use a thin layer on top of that it's a third-party library called ghast it's practically identical to AST but it handles all the version differences between Python 2 and Python 3 it's worth mentioning at this point that autograph operates entirely at ast level there is no lower level intermediate representation and we never interact with the bytecode and that has some unique advantages now the third step there's the bulk of the work and that's quite both literally and figuratively the standard Python library offers the mechanism that helps us with that as well the ASD module provides a mechanism for visiting and transforming a STS that mechanism uses the visitor pattern and it's sketched here basically you get some callbacks whenever the visitor encounters different types of nodes and on top of that we have built an entire library of such transformations as we've seen in the previous diagram these transformations are called in sequence now once transformed the DST is unpowered back into source code in the form of a string there is a standard library for doing that but thankfully there is a third-party library called a store which does a decent job job at that essentially it's lots of string concatenation there's nothing special about that finally the source code is being outputted into a file and then loaded using a mechanism that's identical to writing an import statement once more python helps us with that with the standard module called imp the special thing about imp is that it only works with files on disk hence the need to generate temporary file I should also make a slight note that one other another mechanism that we could have used would be exact and we find we've been going back and forth between using that and imp there are pros and cons to using each so we might revisit this in the future all right a few other mechanisms that are worth mentioning one of them is the templating system that we developed to help us with generating code it essentially lets us write templates in the form of strings code blocks and strings and they support placeholders and they let us generate more complex or new STS if you ever poke inside the transformations library you will see plenty of such templates another important piece is the static analysis which is critical in supporting certain transformations and we'll see more about that in a bit analysis itself can be viewed as just a simple walk over the ASD and it annotates nodes with relevant information another important mechanism is caching caching itself is completely transparent to user but it does help us make sure that every function is converted no more than once loosely speaking the this cache relies on the key assumption that the conversion process is entirely static that is the generated code what ends up in the generated code does not depend on any arguments or variables or any other state of Python basically if you look at some plain code on paper you would know exactly what the output code should be next let's talk about some of the actual transformations that are being made and before I proceed I should clarify that I'll use the word variable a lot these are meant to mean Python variables not to be confused with tensorflow variables which are not involved here all right one such transformation is simplifying the code by replacing some statements with other statements simpler ones such as for instance we replace break statements with variables and additional if statements this essentially helps us avoid the need to to build special handlers in tensorflow for this statements like break they're just easier to lower into something simpler and the process it yes like while you know and less than a million break that's going to be very efficient when inefficient inverted it will still loop over the maximal range it will not look over the maximum range because the wild statement as seen in this example will have his condition augmented so yes the overhead is one maybe two extra conditionals not more than that yes you can see the freshness we have small modules it's not mentioned here agencia we look at the functions Global's it's a closure and those indeed depend on the on the context variables so if you take a function you convert it and you get a certain name and then suppose you create some other function and then you run the converted function it might clash that's very unlikely because if you change the code that we transformed then the function will be reconverted right so there's I'm not sure there's it's even possible to get a clan well you could get a clash in theory but you will have to work very hard to to do that well yeah that's that's a very good observation that's that is one of the deviations from converted entirely static there are some minor exceptions all right so going back to the lowering I'm not going to describe the entire process because it's fairly straightforward I think an example would would suffice for instance here our notice that the break statement which was replaced with a did break variable and then we have a few extra conditionals like for instance this one at the bottom if did not break I start codes to to to protect the the code and the conversions for continued and return statements are similar right another important type of conversion is for function calls we do this for all function calls we place them with a call to a special wrapper this wrapper as it named as its name suggests might decide to convert the function at runtime to convert the target function at runtime but it may not it may decide not to do that many functions don't need to be converted but the important part is that we replace all function calls because statically we do not know what their type is and we do not know whether we should convert them at or not so we defer that decision to runtime one probably were another mention that's probably worth making here is that from the from the graphs perspect the functions are in line we don't in line we don't create any TF function and we don't create any graph functions so from this perspective autograph is consistent with existing with v1 Starcraft code what do you mean by runtime do you intend surf or under or when a Python user runs it that's a very good question there's more than one time or more than one runtime in this case I'm referring to the Python runtime all right next the quintessential transformation we might say is converting control flow so let's look at if statements first the transformation itself is actually quite mechanical for example the body of the if statement becomes a new function and we add some return values to it and we'll see it more about that in a moment and if itself becomes a call to a special autograph overload I'm of course omitting the else block here for for the sake of brevity but its equivalent with the main block main body of the if statement once more all the if statements are converted in this way and here we have an example for a statement noting that note that there's nothing out of the ordinary here the body of the if becomes a body in a new function and the if statement is replaced with the function call now loops are ever so slightly more complicated because the use state variable but not by much once more the block the body of the loop is transformed into a new function this time the function has arguments representing the loop variables the conditional also becomes a function this time because it depends on the loop variables and once more the statement itself is replaced with a function call now the more interesting question is how do we decide which are the loop variables in your program we could of course take all the variables in scope and make them variables but that would be quite inefficient the heuristic we use to do that is actually quite simple it relies on static analysis and in short of a loop variable must be both of these two conditions first it has to be modified inside the loop which is quite evident if the loop doesn't modified then it's invariant to it and the second condition is that it has to be either live in or out of the loop now what life means is live in means that the loop depends on the value of the variable before it enters the loop seems like if something is a new condition it would be it wouldn't have to be live in for a loop these conditions if it's a loop condition then the variable would be live into the loop because it's read before anything else I'll show an example that that hopefully clarifies that a bit the life out of the loop is similar if the variable is used after the loop then it's live out so here we have an example a is to Josh's remark a is both modified by the loop but also live in Tula the loop because once you enter the loop the first thing that happens a is being read so the variable the value of a before the loop is definitely relevant if a starts positive then the loop cycle if a is zero then the loop will not so as live into the loop being on the other side on the other hand is not modified by the loop so we can leave that one out and see is also interesting because it is modified inside the loop but it is not live that's because as soon as you enter the loop the C variable is being immediately overwritten so the fact that it had the value three before the loop is completely irrelevant to the loop because that value is being destroyed regardless and this is a sketch of the resulting code and I'll leave it as an exercise to verify that that it is indeed correct next as I mentioned the conversion process is entirely static and all the statements are being converted all the function calls are being converted and that means that the overloads must handle any type or value verifications a verification at run time once more at the Python runtime yes I guess it's not only things that are assigned a function that's a great observation and I will talk about that in a bit more detail in order for autograph to correctly convert convert code this is only when it transforms the loop into a tensorflow loop the side-effects such as these modifications have to be have to be visible so if you build a function that hides that modification that autograph will not detect it that's it that's an excellent observation I'll get to that I'll get to a specific example of that in a moment all right so as I was mentioning there's the dynamic dispatch that's handled by all operators an interesting observation here is that if we were to convert pure Python code in this way with autograph it would become quite slower because if every if statement will do an is instance or some type track so you can imagine it would be much lower than normal Python code however in the case of tensorflow for our purposes this overhead is peanuts compared when creating jobs so it doesn't really bother us in the case of building graphs all right so far when describing the process I ignored an important piece in Python Python and that is the variable scoping so let's look at an example of that issue with simple conditional which just increments value you know in an if statement now naively copying this block inside a function won't work because due to pythons scoping rule X would become a variable local to the if true loop so any modification that you make to it would be lost to the to the Python runtime in fact you actually get an error here because inside if 2x is a lock local variable by incrementing it you're trying to access an undefined variable so the way we solved that was by renaming the variables inside the the function body and essentially what we're after is avoiding to modify directly the X variable because that's what causes Python to consider it a variable local to the to the state to the to the function now a quick note on unmuted on mutating variables so this what I just showed was valid for simple variables like x equals 1 and so on and so forth mutating them like this statement where we say X dot a equals to something that is fine because that will not cause X or X dot a or anything to become local to the to the function so X still points the correct objects or and when you're safe safe so mutating objects in this case doesn't bother us but unconditionally B data X right that's that's exactly so yes and that's exactly what I'm going into with a bit more detail on these mutations as and that's as Alex alluded you have to be careful about about the effects of tracing when when running tensorflow statement by you means we autograph so we have to handle that case ok so the complication one of the complications with mutation is probably best explained with this simple example suppose we have a method that mutates itself is it does it does some changes in a loop to one of its its properties nothing out of the ordinary with a naive transformation of this loop not doing anything is fine it is correct if that loop was executed as a Python loop and the effective code that executes it looks kind of like this this is not what's generated that while loop at the bottom is actually in fact an autograph overload but effective code that runs is this so we have a loop blue body function a loop conne function and there's a wide loop which calls them as you might expect all this works fine if it's a Python loop however it no longer works fine if it's a tensorflow loop why is that well I'll leave it as an exercise to think what is the value of self dot a as this statement executes and after this TF while loop runs and also what happens to the TF while loop at runtime but I'll just go to a possible solution for this and that is to create a temporary a special loop variable corresponding to self dot a itself and if we do this with a bit of extra of carefully handling so that we put in self dot we have self dot a pointer to the correct value both both inside the body and inside the con and after we have executed the while loop if we do this the TF while loop will execute correctly and you will get the result that you would expect the problem with that is that it breaks Python semantics and that's something we do not want to happen to show that let's consider that a is not a trivial property but it's a custom setter that you defined in your code and let's consider that that custom set or has some side effects for instance it prints some messages now with this equivalent code what would happen when it ran in it ran as a Python loop well there's a lot of assignments there are too many assignments too of that day and that will trigger the side-effects in your custom property so you will see way too many prints in this case and that's definitely something we do not want to happen because we want to preserve Python semantics I'm sorry there was question it seems like this transformation is just wrong that's a very good question if we didn't go ahead and replace all those self days with southland array then if you called any method suppose you called a method that itself behind the scenes did some more modifications to solve Part A then that method would not capture the value of self dot a right so we have to make sure we have to put the proper values inside CELTA inside self dot a because some other code might need it all right so the way we solve this problem is to put this these extra modifications in two separate functions we call them get state and set state because in a way they capture the state of the of the ranta of the execution of the Python execution runtime add during tracing but what these allow us to do is they allow us to do this kind of modifications in the case of the FDF while loops and then in the case of the in the case when we run regular Python loops we just don't call these methods so all the paths are both paths are happy now right next if there are no additional questions I know this is a this is definitely one of the trickier parts of autograph itself but if there are no questions let's go to the second part and talk about the API is that users can interact with first and foremost the absolute recommendation is that if you can use function just use that it has certain advantages it can do automatic control dependencies there are some types of rapey eyes that only work inside Thea function it also caches cassia's the graph in addition to other things and it has additional smarts to handle exceptions so definitely function is recommended but if you really really can't use their function you can call autograph directly there's a specialized the API but keep in mind that it does not add automatic control dependencies and it's also less user friendly other ApS that you can use to tweak things include do not convert annotation which has the obvious effect although even in that case it's still preferable to use TF function with autograph equals false also if you'd like to have a look at the generated code there is an api for that and as we experiment with new transformations there is this feature enumeration that you can use to enable them to enable transformations that are not stable enough to train its introduction now a few words a few notes on on the buggy by far the best way to debug code is to run it eagerly and there is a tense there is a function in tensorflow that help you help you do that or you can just remove the TF function decorator but this one lets you do it without changes to the code and that causes TF functions to run eagerly and you can use B DB and everything else inside there if running legally is not an option than one way to peek inside what autograph is doing is to crank up the verbosity there's this API for that increasing the verbosity is also useful when piling bugs there is a caution I should make of course increasing the verbosity call it can cause quite quite a bit of vlog spam but it will also dump data in addition to code it will log argument function arguments and things like that so please be careful when sharing verbose logs now if you do enable PDP inside autograph code using for instance PDP set trace that will not crash it will work in some way just remember that set trace is a function and like every other functions will be converted with an autograph overload and what that means is that PDB will land somewhere inside graph API at the time of this talk you have to step out twice to get back into the generated code and the other caveat is that of course you will land inside generating code okay now a note on this generated code it definitely contains lots of boilerplate it is designed to be robust not pretty and ideally in a perfect world you should never have to deal with it you should never have to see that code and we're working towards towards achieving that but until then if you do end up in a situation where you have to deal with generated code even if you see it by accident or not or even if you have to actually deal with it please file a bug so that we can work towards avoiding that kind of exposure all right in the next section I want to mention some of the semantics related to autograph because these dictate what you should and you should not expect of it now rather than a detailed detailed explanation I'll just list some broad guiding principles the first such principle is that we intend out autograph to preserve the semantics of existing well behaved code by well behaved I mean in generally it runs without raising an exception so traditional pure Python code should be functionally not changed under autograph and the same should be valid about existing tfv one graph code now to be clear if you give such code to autograph it will transform it but you should not expect its functionality the functionality of the transformed code to change then with respect to eager code autograph obviously supports a subset a bigger but the parts that it does support then those should also preserve functionality as they go back and forth between between eager and autograph that means that if you have that autograph code should not change this functionality when executed eagerly and that essentially is what lets you remove the tear function annotation without having to modify the code in any way so at least in theory when you remove the a function the behavior of the control not change except yes really executing I guess I should have assumed that we just disabled function at the same time we disabled photograph yes you disabled everything basically you run the code exactly as it looks yes there is a flag to turn it off or you could remove the TA functional annotation either of those should not change the behavior of the code that's fine still those transformations it just runs it even it doesn't do the autocrat transformation anymore okay so there is a mode where we can do the automation and then sterber and eagerly there is but not with TF function with TF function is either running graph with autograph or running Iger without autograph yes with that one you could potentially create some graph like code and execute it eagerly and that one should also preserve its functionality but that's that's of course in the eager semantics right ego should the executed graph code as if right no this is just for kicks yeah it's cool but even then keep in mind that wow there's code that runs in graph and there's code that runs in eager so just make sure that those two are truly consistent right yeah all right an implication of these guiding principle is that principles is that code which does not depend on tensor flow objects will not run as a tensor flow statement and that should make it hopefully easy to reason about code so let me show you an example of these four four statements for loops the one at the top is legal Python code it doesn't depend on any tensors therefore it will run as a Python loop the other three will run as tensorflow loops because well one depends on the TF range the other one depends on the data said yeah the third one depends on distribution strategy right in this last last section I'll go through some usage examples which I believe are most interesting from a user's perspective now I will focus a bit on use cases which are illegal in autograph because we have lots and lots of samples of fairly complex code that works but we have fewer examples of code that doesn't work so here they are okay so I'll begin with control flow and as a warm-up I'll show the ideal code for autograph this is definitely code that autograph can handle and it's code that we like most that's because it does its operations in plain sight no hidden side effects no hidden triggers everything is plain another example that works well is using statements like break as we've seen these are lowered so autograph can deal with those finance what now here's an example of code that has certain limitations when running as a tensor flow control control so this if statement it can it depends on a tensor therefore it will run as a TF con however notice that this X variable is only initialized in the in the if branch and not initialized in the else branch and tensor flow as we know does not have the notion of non values or undefined variables so we cannot do it we could not just do this in a TF cond and instead we raise an error and this is actually one of the better error messages that we raised where we explained that you have to initialize the X in the in the else branch so optional value that is true yes and that's why I'm very excited about optionals what if X is local to their branch that is fine yes if it's local then it's fine that's why we go through all that pain to do liveness analysis and what if you just so that local variables don't trip it okay now the same restriction can extend to things that you might not expect for instance if we have a return value that would cause the if statement to deal with an undefined value this is also illegal and the error message is pretty nice in this case as well it tells you that you have to return a value from the else branch as well another example of the same limitation this time dealing involving object properties in this case the error message is a bit confusing and we're working to actually fix it there are messages actually I in my opinion it's very confusing because it's it's a it's a TF cond that actually would execute the else branch but there's no else branch so when is it trying to access a it's definitely a confusing error but it will be much more friendlier hopefully soon okay one quick note that these limitations around the nun or undefined symbols can easily be avoided by initializing your variables early so if you initialize for instance our X at the top with some default value then everything else would work nicely once more so they're fairly easy to prevent now if I could be pedantic for just a moment if I would like to recommend that when you have to deal with situations where you have some default values I definitely recommend that you have a separate very four separate boolean to represent the state of not initialized or not value or not valid I definitely recommend that over using magic values doing that can save your world of pain later on and this is not strictly it's it's totally unrelated to auto craft it's just a recommendation of a good practice in general okay now a more significant limitation in tester for control flow is around hidden side effects as we actually had a question alluding to this so let's look at this example where we have a simple helper method that mutates the state of self then there is another method that calls this helper so when converting this larger method this method F autographed static analysis when looking at the variables for that if statement has no way of seeing that self dot T is being modified because that's hidden inside the method and we do not do cross function analysis well not yes not yet at least so that means that the TF count will ultimately end up not accounting for that modification to self dot a and you get this rather obscure error message which is quite confused once more the error message can and hopefully will be more helpful but the limitation itself remains solutions are definitely possible and I think they would make a nice nice future project but for now they are a matter of future development now a good defensive against these kind of kinds of patterns is to use a more functional style in in your code functional style that means if you if your function modifies the value returned it and that that helps autograph for instance in in this example if we modify our code like this to return the new value the new value that should be put in subtlety and then we do the assignment in plain inside the converted function then things are happy once more and everything works and this is my last bit of pedantry I hope for this for this talk in general functional style tends to be loved by by machines compilers and static analysis and analyzers have a much easier time dealing with functional code code that takes its inputs as arguments and returns everything that that's modified and sometimes it could make it could help the code become more readable - all right next up a few examples around data sets which are quite quite satisfying in my opinion because it shows that the underlying T of data and this strategy api's are powerful enough to facilitate this kinds of conversions so the first example is that iterators the TF data iterators work in almost all cases and we'll see the few exceptions in a second but essentially you can take the iterator you can go through parts of it and then you can break out of the loop and then you can resume it and everything works as you would expect and if you're curious the implementations for for loops using TF data iterators and data sets are actually they might actually be quite they're quite an interesting read I think they're quite a nice feat the code with all those callbacks might be difficult to follow but in my opinion it's quite interesting and you can find it in the in the specialized control for operators of autograph especially for datasets we actually to handle a for loop we end up applying three operations in sequence scan take while and reduce and I think that's this pretty pretty nice alright consuming an iterator with the next function also works and just to be clear this is code that runs as graph code alright next let's talk about handling runtime exceptions and since we were just talking about iterators let's talk about the common pattern in Python it's generally considered pythonic such a pattern where you just try things and catch an exception if they don't work so instead of having a pattern where you say if I can do foo do for this pattern is do for try to accept can't do it right and one of the most common uses of such a pattern is the use of iterators where you have a loop and inside the try except block you just try to call next now for for pure python control flow under autograph this works just fine it works the way you would you would expect unfortunately that doesn't work for tensor flow control flow and that's because well it's a dual reason one of them is that there is no exception catching in graph mode there is no op that catches tensorflow runtime exceptions now on the other hand we could conceive that autograph could lower exceptions I mean we lower return statements therefore we should be able to lower exceptions as well however that could make the code prohibitively complex and slow for instance because any statement could conceivably raise an exception you would have to wrap each line with an if statement so the lowering would not look very pretty at least not in the trivial case okay so the implication of this means that if you have a tensor flow control flow statement and you wrap wrap a next call into such a try except block autograph will not complain about it it will it will leave the try except into the code it will not transform it but if you think about it the effective code will completely ignore it the runtime terms of exception since there is no caching inside the graph any runtime error will bubble all the way through the through the tensorflow runtime so it's just essentially what this means is that if you put a try except inside graph code the exception will not be caught it will just fly past the accept statement it also means that if you do end up trying to catch exceptions you should do that in eager mode outside of the TF function because integral mode you can catch them right the runtime exception bubbles through the tensor flow runtime and then it's captured by TF function and we raised as as a Python exception so you can't catch it just not inside the graph should we add try catch up to them I would love that I don't know what the implications about optimization and so they are I think it's it's a little scary because of the unknown semantics of what ops could be running in parallel at the same time one of them generates an error so it's do those cake the other ones get canceled or what that's true we can't actually use actual exceptions to this building without oh we wouldn't use your first pass of exceptions right we're gonna have to be a new tensorflow runtime language feature except some tensor that takes three function bears and calls one calls another of the first one raise an exception and then always calls the third one she's like simulate the behavior of try catch finally that would have the consequences pointed out but their cancellation so it's very non deterministic about what actually gets executed in the presence of an error but yeah it's a separate discussion we've definitely stirred the hornet's nest or this loop is a tensor flow that's one that's an excellent question in this particular case and just implying that stop condition returns a tensor in general we look at the condition of the loop if the condition object is a tensor then it will be a tfy loop if it's a Python boolean or whatnot then it will be plain Python loop and it will be unrolled does this answer your question all right okay so one going past adding a tf-x catch exception to tensor flow now there is a bit of a there is there are good news about this in that is that in most of the code you can avoid having to catch exceptions altogether for instance with data sets you can transpose the computation a bit and instead of having a while loop you can have for loop over the data set and that will make sure that the loop stops when the data set is exhausted and then you move the condition inside the if statement because we do support break statements so these two pieces of code the one in the right and one in the left are functionally equivalent and if you squint I think it's actually even shorter so I might if I daresay it's actually cleaner all right next up let's discuss a bit about collections so again in normal pure Python code this code this snippet is as you probably expect it's it's very common where you just take a list and operate on it that we do not invite on a lot right and as with any other pure Python code this works just fine under autograph but if the loop is a tensorflow loop things don't work anymore and once more you get a rather obscure error message that we will hopefully for fixed soon but in general the loop is that you cannot operate on a Python collection from tensorflow control flow that just doesn't align with with things and it's not not supported by auto crap instead it's a good practice to use specialized tensorflow collections like for instance tensor array and it's a good idea to do it even when you work in legal mode because that would mean that you don't have to modify the code if you want if you ever want to go to 202 autograph the problem with that transformation is that it's difficult to do retroactive the main problem if we look at this back slide there is this L being initialized with an empty Python list first and foremost we don't know the type of that list and we don't know at this line it's unclear whether the user even wanted a Python list or a tensorflow list so we'd be forced to make assumptions that would violate the semantics of normal Python code it's definitely definitely a challenge that's why we resorted to the rule that okay if you want a tensorflow list please be explicit about this and we will offer as much syntactic sugar around that as possible but you have to explicitly request it all right let's see a few other examples that that need special attention this time around routes that change the type or shape of their variables so first off a probably familiar example of a type tester for a while loop which you probably are already aware that tensor of all limits the degrees to which tensors can change shapes or D types inside the loop itself and you typically get error message of this kind that some tensor enters the loop either strip and has a different shape after one iterations one iteration now there are ways to deal with this one of which is specifying shape invariants for the loop and we're working to add support for for that in autograph as well with a special function call that lets you specify them another thing that we're working to address is making the error message more useful as well for example here it would be nicer if the error message was saying something about the variable a rather than some obscure constant zero right a more subtle effect of changing types in a loop is shown here this variable a starts as plain Python value and then inside the loop it becomes a tensor now according to autographs dispatch loop when we first execute the loop when we execute the first iteration it would appear that it's a Python loop because a is a python scalar scalar therefore button loop but after the first iteration it would appear that the loop is in fact tensor for group so we're working to improve the error message here as well to be explicit that that happens but in the future we hope to actually just deal with that directly so for instance you can that we could cycle through the loop a couple of times and if we decide that it should that the Python hook should become a tensorflow loop we should just do that and then we would only have a few unrolled iterations before the loop all right now going back to exceptions and to errors let me show you a few examples of how autograph modifies them so that they don't point to generated code now first graph construction errors are being modified by expanding their error message that is purely the the error message itself we don't we don't touch the trace back of that error that will still point to a generated code however the error message has this stack trace like message that helps you locate the cause of the error in the original source code this is if you think about it this is very similar to the way tensorflow runtime errors have a second stack trace showing you the location of where the OP was created now with this this documentation of the error message is done using a mechanism that wraps the original exception into a new one and unfortunately we don't have time today to discuss a lot of details about how it's done but what's important to mention is that most of the time the type of the error does not change so if you raise the value if if the tensorflow runtime sorry if the if your app raises a value error then the users will see a value error as well it's just that it's message will be changed however sometimes when we cannot replace errors using the same type you might see staging error instead and that typically handle happens when you have constructors that are complex and we cannot which we and we cannot figure out how to call the construction the call the constructor in a way that keeps the gives the data most errors just have a simple init constructor with just a string message and there we can just create a new error of the same type with an X the message but where we can't do that you will see this staging error the original type of the exception can still be recovered so you can still inspect the exception to find the my special exception type and its original message and so forth okay now lastly runtime errors are modified in a similar fashion so that they don't point to generated code in this case we simply the the error message is not actually changed from what was originally raised we just simply replace the references so here you will see I ran this code in ipython and you see that's why you see this ipython input that's the reference to the cell what's important is that you don't see a reference to some temporary file that contains generated code all right now once more I probably repeated that quite a bit in the talk we try as much as possible to remove the references to generated code from error messages if you do see any messages or an where that's still not the case please do file a bug okay one last example that I want to show is how decorators are handled and in general in Python decorators are just syntactic sugar they are just high order functions that get executed when the code is loaded from many perspective that's that's why the creators are actually difficult to detect because they they're not not you realized in the ASD at least most of the time they're not anyway when you convert decorated functions with autograph the decorator will be converted and that's the reason why you will usually see the source code of the wrapper for instance if you have this decorator that just replaces the function with a wrapper and you try to convert if you try to convert the decorated function you will see the source code of the wrapper instead that should be no cause for alarm because the recursive conversion will end up will step into into the wrapper and convert the target function as well so things do work as as expected all right that's it what I that's it for now that's all I had for today there are a lot of other topics that I didn't cover for lack of time and I hope these and many other examples would be discussed in a more in more detail in a more comprehensive reference guide that is in currently in the works and that's it thank you very much [Applause]