Swift for TensorFlow - TFiwS (TensorFlow Dev Summit 2018)

Key Takeaways

[Music] hey there everyone I'm excited to introduce a new project we've been working on that takes a new project to improve usability of tensorflow and we care so much about usability here that we're going all the way back to first principles of the computation that we're performing but first why usability I hope that everyone here agrees the the productivity and machine learning is critical because it leads to a faster pace of innovation in progress in our field and of course we just want to build beautiful things for tensorflow users and so that's a big piece of it as well but if you look at machine learning frameworks there's two major approaches the most familiar are the graph building approaches where we explicitly define a graph and then execute it turn your computation this is great for performance but isn't always the best for usability in contrast the defined by run approaches like eager execution aren't always the best for performance but they get they allow you to use language control flow and other things that make them a lot easier to use the interesting thing about both of these approaches is that they're really about allowing Python to understand the difference between the tensor computation your code and all the other non tensor stuff like command line often processing and visualization and the other things that you do and so I think it's interesting to look at how these actually work in the case of eager execution for example you write your model and Python starts out by parsing it and then it feeds every statement at a time to the interpreter and if it's a tensor operation it fires up tensorflow and they hands it off intensive flow takes care of your tensor operations otherwise the Python interpreter runs it now the key thing about eager execution and graph building is that they're designed within the constraints of what you can do as a Python library now if you crack open the compiler and look at it from what could we do with the compiler and language involved there's a whole other set of approaches that can be applied to solving this particular problem and that's what we're doing the cool thing about a compiler is that after you parse your code the compiler can see the entire program and all the tensor offs in it and so what we're doing is we're gonna add a new stage to the compiler that automatically takes these tensor operations out of your program builds a standard tensor flow graph for you and then hands it to tensorflow and because it's a standard tensor flow graph you get full access to all the things that tensorflow can do including all the devices now we think that this approach is a really great combination because you get all the power and flexibility of tensor flow but you also get the usability of your execution as well but there's a catch there's always a catch right the catch here is that we can't do this with python at least not with the raw reliability we expect because it doesn't support the kind of compiler analysis we need and what do we mean by that well the compiler has to be able to reason about values as they flow through your program have some reason about control flow and function calls have some reason about variable aliasing and other things like that if we can't directly use Python then as users of course we care a lot about all the nice things we've come to love about Python including the ability to use all the standard Python API s so I know what you're thinking does this mean that we're talking about doing a new language well that's definitely an approach to solve the technical requirements we want and if we do new language we can obviously build all the nice things we want into it but this comes at a cost because it turns out that we'd be forgoing the benefits of having a community that includes things like tools and libraries but also things like books which some people still use and even more significantly this would take years of time to get right and machine learning just moves too fast no we think it's better to use an existing language but here we have to be careful because to do this right we have to make significant improvements to the compiler and the language and we need to be able to do this in a reasonable amount of time and so of course this brings us to the Swift programming language now I assume that most of you are not very familiar with Swift so I'll give you a quick introduction Swift is designed with a lightweight syntax and it's really geared towards being easy to you to use and learn Swift draws together best practices from lots of different places including things like functional programming and generics swift builds on LVM which means that of course it has an interpreter and it has a scripting and capabilities as well Swift is great in nope environments and these are really awesome when you're interactively developing in real-time swished is also open source it's portable for lots of platforms and has a big community of people but the number one thing that's most important to us is it has an open fully open design environment called swift evolution which allows us to propose first-class machine learning language and compiler features directly for integration into Swift and if you bring all of this together we've been talking about here I'm happy to introduce Swift for tensorflow swift for tensorflow gives you the full performance of graphs you can use native language control flow has built-in support for automatic differentiation it could detect errors in your code without even running it it has full access to Python api's but instead of telling you about it I'd like to invite up Richard wave to show it to you now thank you Chris I'm thrilled to show you Swift for tensorflow Swift is a high-performance modern programming language and today for the very first time Swift has a full power tensorflow built right in I'm gonna walk through three major styles of programming scripting interpreting and notebooks so first let me show you the Swift interpreter this is a swift interpreter when I type some code so it evaluates it and prints the result just like Python now let's import tensorflow I can create a tensor from some scalars now I can do any tensorflow operation directly and see the result just like I would with eager execution for example a plus a or ACE matrix product with itself of course loops just work I can print the result of a now interpreters a lot of fun to work with but I like using tensor flow in a more interactive environment just like Jupiter notebook so let's see how they work this is a Swift notebook it shows all the results on the right so here's a more interesting code fun with functions so here I have a sigmoid function inside a loop now as I click on this button it shows that a trace of all values produced by this function over time now as a Mishler in developer I often like to differentiate functions now when I type in well since we were able to improve that programming language we build first-class automatic differentiation writing to Swift now when I type in gradient effects it shows the gradient so it computes the gradient automatically and gives me the result so here's the gradient a sigmoid now let's look at some Python code let's think about Python well as a michelin developer I've been using Python or and I know there are many great Python libraries just today my colleague Dan sent me a data set in picault format well I can directly use Python api's to load it all I have to do is type in import icon and swift uses a Python API pickle to be supposed to be specific to load the data in here you can see the data ride in the Swift notebook now so here's the Swift notebook now some people like to run training scripts directly in command line so let me show you how to train a simple model from internet line so here here is a simple endless model in this model I'm using tensor flows data set API to low training data as I scroll down I have the four pass backward pass gradient descent all defined in the training loop now I usually like to work on the go so this code has been working on the CPU on my laptop but when I want to get more performance what do I do well why don't I just enable cloud TPU so all I have to do is add one line to enable TPU execution when I save this file open the terminal to run this training script it's initializing TPU and the Swift compiler automatically partitions is programmed into a normal suit program and a tensor flow graph and tinsel flow is sending this graph to the tensor flow SLA compiler for TPU execution now it's running and we're waiting for the TPU key to give the result look the losses going down all right so why don't we simply open pin support and see the training curve so now I can see the entire training history in tensor board so this is looking great now this is Swift for tensor flow it's an interactive programming experience with supercomputing performance at your fingertips back to you Chris thanks all right to recap quickly Richard show you the Swift has an interpreter and it works just like you'd expect now I know that it's super frustrating to be working on a program and two hours into a training run you get a shape error or a type mismatch so if tis designed to catch errors early and we've started building in support for catching tensorflow specific mistakes right into tensorflow we showed that it's you can directly use arbitrary Python api's and other dynamic languages directly from Swift which gives you full access to the data science ecosystem and any other Python API is that you just love to use Swift is also during standard tensor flow graphs including control flow which give you the full performance of the session API and of course graphs are also awesome because they give you access to everything that tensor flow can do including devices spanning the range from the tiniest raspberry pi all the way up to a TPU supercomputer now you may be wondering when do you get this what does this mean well this is still an early-stage prospect but I'm happy to say that we're looking forward to our open source release next month and not only are we releasing the code we're releasing technical white papers and documents explain how it works and we're also moving our design discussions out in the public on to a Google group so that everybody who wants you can participate now we're not done yet so we have basics for support for automatic differentiation built right into the compiler in the language but we're also really interested in extending it just for exotic cases like recursion and even differentiating custom data structures compatibility issues are also super frustrating particularly if you accidentally use an OP or a d type that's not supported by your device Swift has great support for detecting availability issues like this and we're looking forward to wiring this right into support tensorflow we're also interested in high-level api's but with this we want to be a little bit cautious we have some prototypes now but we'd really like to work with the community and evaluate and iterate and design multiple different approaches and actually experiment with these and then settle on the best one based on real world experience now this has been a super quick tour of Swift for tensorflow Swift for tensorflow combines the power and flexibility of tensorflow with whole new standard for usability we think it's going to take your productivity and shoot it to the roof so if retentive flow is also an early stage project and so we'd really love for you to get interested and help us to build this future thank you [Applause] [Music] [Music] you