Eager Execution (TensorFlow Dev Summit 2018)

Key Takeaways

[Music] hello my name is Alex and I'm here to tell you about eager execution which I think you've heard in the last two talks now but I'm here to tell you what it's actually about this new like imperative object-oriented pythonic way of using tensorflow that we're introducing to you today as part of tensorflow core so you know because you're here or because you're watching the live stream I hope that tensorflow has been this like graph execution engine for machine learning that lets you run graphs like high scale and all sorts of other nice things but has it and why did we choose to go with graphs in the first place since now we're I'm gonna tell you about eager execution where we moved beyond what we can achieve with graphs I think it's a good idea to recap why we bothered and like a really good reason why you want to have your computation represented as a platform independent graph is that once you have that it's very easy to differentiate that graph and I I went to grad school before altered if was standard in machine learning toolkits and I do not wish that on anyone right its life is much better now trust me also if you have a platform independent abstract representation of your computation you can just go and apply it to pretty much anything you want you can run it on the TPU you can run on a GPU you can put it on a phone you can put it on a Raspberry Pi there like all sorts of cool deployment scenarios are gonna hear about today and this is it's really valuable to have this kind of platform independent view also compilers work with data for graphs internally and they know how to do all sorts of nice optimizations that rely on having a global view of your computation like constant folding common sub-expression elimination like data laying and things like that and a lot of these optimizations are really like deep learning specific like we can choose how to properly layout your channels and your height and width and stuff so your convolutions are faster and finally like a key reason that's very important to us at Google and I hope important to you as well is that once you have a platform independent like representation of your computation you can just deploy it and distribute it across hundreds of machines or a TPU pod like they showed earlier and this is a very seamless process so these graphs are so good what made us think that now it's a good idea to move beyond them and let you do eager execution good place to start is that you don't actually have to give up automatic differentiation I'm sure like you're familiar of like other frameworks like pythons autograph that let sorry Ulta grad they let you differentiate dynamic code and you don't need to have an a priori view of your computations to differentiate it you can just build up a trace as you go and then walk back the trace to compute gradients also if you don't stop to build a platform and like this computational graph you can iterate a lot more quickly you can play with your model as you build it you can inspect it you can poke and prod in it and this can let you just be more productive when you're like making all these changes also you can run your model for debuggers and profilers and add all sorts of like analysis tools to them to just really understand how they're doing what they're doing and finally if we don't force you to represent your computation in a separate way then the host programming language you're using you can just use ultimate like machinery of your host programming language to do control flow and data flow and complicated data structures which for some models is key to being able to make your model working at all so I hope you're not wondering how do I get to use this and the way you use this is super easy you import things to flow and you call TF that enable eager execution and once you do that what happens is anytime you run a tensor flow operation like in this case a matmo instead of tensor flow building a graph that later when run when executed is going to run that matrix multiplication we just immediately run that matrix multiplication for you and give you the result and you can print it you can slice it you can dice it you can do whatever you want with it and because things are happening idiot lis you can have highly dynamic control flow that depends on the actual values of the computation you're executing and here is just a simple like wolf conditions line search example that I wrote and it doesn't matter it just matters it has like while loops that depend on like complicated values are computed based on the computation and this runs just fine on whatever device you have and together with this enable eager execution finger we're also bringing you a few symbols intensify that make it easier for you to write code that's going to work with both when building graphs you know executing eagerly and we also do bringing you a new way of doing gradients because I'm sure you're familiar now with how you do gradients in normal terms of all where you just create your variable you create your last function and I hope you can think of a better loss function than this one and then you call T after gradients to like differentiate it but when you have eager execution we try to be as efficient as we possibly can and if you're going to one thing you think about is if you go into different shader computation you need to keep track in memory of information about what happened so far like your activations and things like that but I don't want you to pay for the cost of this tracking when you're not computing gradients because performance is really like the whole reason why we're doing this is because we want to use these big nice pieces of hardware that train models super fast so in equally your executions enabled when you want to compute gradients you use this little context manager to keep a tape active and the tape just records all the operations you execute so we can play it back when you compute the gradients otherwise the API is the same also writing training loops in eager as Derek pointed out is much if it's very easy and straightforward you can just use a Python for loop to iterate over your data sets in datasets work in eager just fine and they work with the same high performance you get in the graph execution engine then you can just do your predictions compute your gradient supply your gradients and do you know all the things you're used to doing but really the interesting thing about eager execution is not when you're just writing this code that's finished that is done that we already know works but when you're still developing when you want to do things like debug so when eager execution is enabled you can just take any model code and I used my simple silly gradient example here add notes to like drop into the Python debugger anywhere you want and once you're in the Python debugger you have the full power of debugging available you can print the value of anything so you can change the value of any tensor you can run any operation you want on any tensor and this will hopefully empowering you to really understand what's going on in your models and really be able to fix any problems you have you can also take eager execution code and profile it using whatever profiling tool you were most familiar and comfortable with so here I have a little domed model that just does a math mole and a bias ad and let's pretend I don't know which of these operations is gonna be the bottleneck which one is lower and I'm sure you all know the answer that the Matt mole is a lot more expensive but here you can just run your code for your Python profiler like you would do with any other programming job and find out that the Matt mole is like 15 times more expensive for my batch size here than my bias addition and also by the way those examples are run on the Google collaboratory thing which is this completely public shared GPU capable interface for Jupiter host book notebooks that are like hosted in Google Cloud it's pretty cool and I think we have a demo on eager that's hosted on that you can play out with later or if you're on live stream you can play out of it now if you can find the link but together with eager we're bringing you a lot of new api's that make it easier for you to build answer photographs and to execute models and these api's are compatible with both ego execution and graph building so one that's been like a recurring low priority feature request is how to customize gradients in tensorflow and I'm sure you're familiar with like a few of the tricks that people have like stop gradients and functions and things like that but we're introducing a new API that works in both ego and graph execution and what I like about this example is that it's a thing that's being asked by many many people how to do if I want to run my forward pass and then in the backward pass take the gradient of a particular tensor and clip it a clip it's norm to keep it small to prevent it from exploding and it just takes six lines of code to make a version of TF that identity that in the backward pass Clips its gradient and I think this is really cool and I look forward to seeing what you guys can do with this when you're doing more than six lines of code and solving like all sorts of new interesting research problems a big big change when programming with eager from graph that I really wants you to stop and think about is that we're trying to make everything as pythonic and object-oriented as possible so variables intensive though are usually a complicated thing to think about but when your execution is enabled it's much simpler a tensor for variable is just a Python object you create one you have it you can write you can change its value you can read this value when the last reference to it goes away you get your memory back even if it's a GPU memory so if you want to share variables you just reuse those objects you don't worry about variable scopes or any other complicated structure and because we have this like object-oriented approach to variables we can look at some of the API is intensive flowing like rethink them in a way that's a little more object-oriented and easier to use in a very one that like really stood out to us is needing like an overhaul as a metrics API so we're introducing this new TFE dog metrics package where each metric has two methods one that updates the value and one that gives you the result and hopefully this is an API that everyone is gonna find familiar to use and please don't try to compare this to the other metrics api we're also giving you a way to do object oriented saving of tensile file models if you've tried looking at test flow checkpoints now you know that they depend on variable names and variable names depend not just on a name you gameshow variable but on all other variables which are present in your graph this can make it a little hard for you to save and load subsets of your model and really control what's in your checkpoint so we're introducing a completely object-oriented Python object based saving API where you it's like python pickle like any variable that's reachable from your model gets saved on your model get saved you can save any subset of your model you can load any subset of your model you can even use this TFE the checkpoint object to build things you want to save that have more than a model and here we have an optimizer and their global step but really you can put whatever you want in there and the idea is that this object graph that eventually sir goes down to variables is something you can save in load so you can have your gain and save and load your discriminators in your generator separate from each other then you can take a discriminator and load it back up is like another neural network that you can use in another part of a model and this should give you a lot more control to get a lot more out of like then for checkpointing but a real question that everybody asks me when I tell them at a work at legal execution is is it fast because graphs have this high performance promise so how fast can we make this thing that runs Python code all the time and the answer is that we can make it fast enough for models that are highly computationally intensive you pretty much don't see any Python overhead and we're as fast as graph tensorflow for sometimes slightly faster in reasons that they don't fully understand even for highly dynamic models you have comparative performance of anything else you can find and please don't get attached to these numbers you have many more benchmarks in our code base and we're optimizing Iger performance very aggressively but I hope that the message you get out of this is that if your model can keep a GPU busy if you're doing large convolutions large matrix multiplications there is almost no cost in experimenting and doing your research in model building with eager securin turned on but when you are doing smaller things there are some overheads and I want to go over them but again don't get attached to them because we're being very aggressive about optimizing this if you just run a no op intensive flow like an identity it takes almost a microsecond to execute it if you run that with eager execution turned on there's an extra microsecond of overhead if you're tracing gradients there are another three microseconds of overhead that you get but if you're just in queueing something on the GPU stream that alone takes like single-digit microseconds so if you can execute enough computation to keep a GPU busy you're unlikely to see anything bad from using your execution and again these numbers are improving very quickly please don't get too attached to them but there is this large ecosystem of tensorflow code libraries models frameworks checkpoints that I don't think anyone wants to give up and I don't want you to give up if you want to use eager execution so we're also thinking really hard about how can you how we can interoperate between eager and graph one way is to like call into graphs from eager code and you can do that with TFE to make template which has this create graph function argument when you pass through to it would build a graph for that little Python function that you and then you can use it in manipulating cartograph from eager execution we also have the reverse which is how to call into eager from a graph let's say have a big graph that you understand everything in it but there's a little chunk of your computation that you really don't know how to express in it either you don't know or you don't want to bother expressing it in using like tensile photographs so you can wrap it in a TFE Python and what you get in there I mean the Python function is executing our eager tensor so you can run any convolution anytime to falafel in including convolutions and other things that are not available in Empire but you can also look at the values and inspect and use dynamic control flow in there so I hope with these two things together you can really reuse eager and graphical across but really the easiest way to get eager and graph compatibility should just write model code that's gonna work in both ways and if you think about it once your model is fully written debug and test there's not much there that tells you whether you need to build a graph or to execute eagerly it so right iterate debug and eager and then import that same code insta graph put in an estimator deploy it on a TPU pod deployed on the GPU distributed into whatever you want and like this is what we've done in our example models and there's gonna be a link in the end of the presentation so you don't need to like worry about writing this down so here's some factual advice for you write code that's gonna work well when executing eagerly and when building graphs and to do that use the Kara's layers they're great they're object oriented they're pythonic they're easy to understand manipulate and play around with use the Kara's model just stitch those layers together that will get you saving and loading and training and all sorts of things automatically if you want but you're not forced to use those used TF contribs instead of TF semi they will move to the tensor board open source package very soon so if you're watching this on video we probably already happened use the TF e matrix instead of the TF matrix because these are object oriented friendlier to use and friendly and eager and use the object based saving which is a much nicer user experience anyway so I hope you're gonna want to do this all the time if you do all of this it's highly likely your code is gonna work super well in both eager execution and graft building so now I'd like to take some time to tell you why you should enable eager execution and like a real good like important reason for us that led us to build this in the first place is that if you're new to machine learning or you're new to tensorflow and you want to learn being able to play these objects and manipulate them directly is just a much nicer experience than having to build a graph and interactive later in the session it's a lot more intuitive lets you understand what's going on much better so if you're new but just by all means go straight into your execution play around of it and figure out how to like get graphs later also if you're a researcher and you're quickly iterating over models you're changing their internal properties and you're comparing them and you try to do models are non-trivial that we in the tensor field team we're not thinking about when a design tends to flow if your execution will make it much easier for you to understand what's going on to debug what's going on to like be productive in advance so if you're a researcher this is for you also if your model is not working and you want to understand why being able to enable eager execution and then step through it in a debugger change some values play around of it understand that is priceless and that has saved me a lot of time similarly if you want to profile your model using like the full power of whatever tool you like to use the profile Python if your execution is your friend also there are some models like recursive Orion ends there are just much easier to express if you don't need to put your entire computation in a static data flow graph if you're working on one of those models eager execution is also a choice for you but really the reason I think you should enable this is that it's fun it's a very nice and in like intuitive way of interacting with tensorflow and I hope you're going to have a lot of fun experimenting with it so now I would like to appoint you a few things some of my colleagues sitting over there now they're gonna be in the demo room during the break with laptops with collabs that I like Jupiter notebooks to let you type and try eager mode there please go give it a try or if you're watching this on the live stream type that short link hopefully it will stay in the screen long enough for you to type it and play with it right now it's really nice we have a getting started guide on tensorflow this should be live now programmers guide slash eager that tells you what initiative are eager execution and what you need to know about starting to use tensorflow using eager execution we also have a ton of example models like from RN ends to confidence through all sorts of things that are available behind that link and I encourage you to look at them and see how it easy it is to write the model and how easy it is to also reuse the same code from graphs for deployment we have deployment for graphs or all models except for the highly dynamic ones that are just really hard to write in a graph form and give it a try if you give it a try let us know how it went we're super excited to share this with you I hope you're gonna have a great time playing with this and yes thank you [Applause] [Music] [Music]