What’s new in TensorFlow 2.11

Key Takeaways

hi do you want to hear the latest updates from tensorflow well you're in the right place I'm Mark from the tensorflow team stay tuned to find out what's new in tensorflow including the latest 2.11 release we've got updates to attention layers detenter decision forests and even some tutorials let's check it out you're probably familiar with the Keras optimizers maybe you've used atom or SGD to train your models well starting from tensorflow 2.9 we have been carefully upgrading this API and introducing new optimizers in tensorflow 2.11 we've turned them on by default don't worry your existing code will still work the same for most users but now you can take advantage of new features like weight Decay and a brand new optimizer adafactor you might find Ada factor using NLP or large language models it uses less memory as it only saves partial information from previous gradients to recap the migration in 2.9 we added new optimizers into the experimental namespace in 2.10 we added the existing optimizers into the Legacy namespace and now in 211 we have turned the new optimizers on by default for everyone you can find more info on the new optimizers in the video description and check out the release notes for more info on how to migrate let us know how you go in the comments or file bugs on GitHub if you're building Transformer models we've made some improvements to the attention apis that should make your life a little easier when building an auto aggressive model you need to hide future training data so that your model only sees the outputs generated before the current time step otherwise it can learn from future data that won't be available at inference time using a mask this is called causal masking and is now automatic in Keras by enabling use causal mask when calling the attention layers Keras will add a mask that ensures that each location cannot attend to the locations in the future we've updated our machine translation tutorial to show this layer in action and you can find a link in the video description or search for it on tensorflow.org the attention layers now support implicit masking too when using the embedding layers enable mask 0 to ensure that any zero padded sequences are masked automatically and The Mask will be propagated through the model you can find an example in the same tutorial tree based models are excellent when you're working with tabular data they're faster trained accurate and interpretable our tree based framework tensorflow decision forests is now version 1.0 this Milestone means the tensorflow decision Forest is mature stable and ready for production use it also brings some new features including native support for tensorflow serving performance improvements and a preview of inference apis for JavaScript and go to learn more about tensorflow decision forests check out the links in the video description detenter provides a way for you to distribute the training of your model across devices to improve efficiency reliability and scalability tensorflow 2.11 packs a whole bunch of detenter updates to speed up and simplify your distributed architectures there's a new TF data data set wrapper to convert your existing data sets so that they emit detensors it supports automatically and efficiently packing the tensor components to devices based on the supplied layouts we've simplified some apis too we now have one function call for initializing any accelerator instead of separate apis for TPU and GPU and detanser now supports checkpointing via TF train checkpoint so you can use the same apis the classic tensorflow training process and get the benefit of efficient Multi-Device save and restore without any API change and we also have a performance boost we've turned on optimization for gpus and CPUs that combine independent all reducers we expect this to speed up small frequent gradient updates across devices and experimentally we have seen pretty big improvements while training bird tensorflow 2.11 introduces warm start embedding Matrix a Keras utility function that makes it easy to update your embedding vocabulary and continue training without missing a beat we've added a new guide to show you how it works search for warm start embedding on tensorflow.org be sure to check out the new Keras group normalization layer not to be confused with batch Norm where stats are computed across the whole batch or layer Norm where they're computed across the whole layer group normalization splits the incoming channels into groups and performs normalization independently within each Channel group group normalization has been shown to help when training with small batch sizes where batch Norm is less effective 2.11 also brings a new experimental structured tensor a collection of structures with the same schema it's suitable for storing tabular data like data frames or structured data like protocol buffers and lastly effective with the upcoming release of tensorflow 2.12 the tensorflow 1 estimator and feature column apis will be considered fully deprecated in favor of their equivalents in Keras we've updated the docs and you will soon start seeing runtime warnings follow the migration guides to get your code up to date tensorflow 2.11 will also be the last version to support python 3.7 this aligns with numpy's python support so make sure you jump ahead to a more recent python release to keep using tensorflow on top of all these exciting features our Doc's team has been busy with some new and improved tutorials I told you earlier about some updates to the Keras attention layers that make masking easier when building Transformers to see these in action and learn how to build your own transformer model check out the newly updated neural machine translation tutorial on tensorflow.org in this tutorial you'll build a machine translation model from scratch using the Keras apis you will learn how attention works and build self and cross attention layers you'll build up the encoder and decoder layers train the model and test it out all in Google collab we've also got a brand new tutorial that shows you how to implement question answering in a mobile app in this tutorial you'll use tensorflow Lite to load a pre-trained Bert model into an Android app then in the app you can pass a document and a question and get back answers all with the native easy to use API that will be familiar to Android developers all right that's enough for this video we've got more detail on these changes in the 2.11 blog post so check that out by the link in the video description or head straight to blog.tensorflow.org to try these new features out head over to tensorflow.org install and grab tensorflow 2.11 for yourself let the team and I know what you think of the latest updates and what else you want to hear about in the comments or over on the tensorflow Forum we'll be back with another update soon make sure you subscribe to stay informed