AI-powered infrastructure: Cloud TPUs

Key Takeaways

[Music] hi everybody Welcome to my session I'm going to be talking about AI powered infrastructure specifically Cloud tpus for the next 10 minutes Cloud TPU tpus stand for tensor Processing Unit it's one of the fastest most efficient most sustainable ml systems in the world designed by our engineers at Google Cloud TPU is one of the key differentiators not just for gcps but for the entire alphabet and here is why many of alphabet's Key Products includes search YouTube translate and Gmail as well as te of-the-art Google AI models such as Lambda Palm Gemini they are all powered by tpus externally we are rapidly growing an ecosystem of ml practitioners who are actively utilizing tpus to put the boundaries of AI in a very cost efficient and sustainable manner importantly our customers can now enjoy a much better ease of use with the recently landed TPM architecture and ml Frameworks of choice the landscape of AI has dramatically shifted larger more complex models have given rise to Greater AI capabilities across a range of applications but at the same time larger models are also driving up the costs for AI Computing astronomically today's state-of-the-art llms are trained on tens of millions of dollars have hundreds of billions of parameters and need trillions of tokens of data to learn at Google we are no stranger to this phenomenon almost a decade ago we realized that if users search their voice just for three minutes per day Google would have to double its Data Center capacity to run speech recognition AI it's with that realization that we built the first generation Cloud TPU since then we have rapidly Advanced with every iteration from purpose built AI chips to the highest bandwidth Optical interconnects to efficient but highly scalable liquid cool data centers tpus have become powerful supercomputing platforms to meet the computational demands of the modern geni workloads let's take a closer look with the latest generation which is our Cloud TPU v5e it's the most cost efficient versatile and scalable Cloud debut to date first is cost efficiency Cloud tpv 5e provides Two Times Higher puff per dollar when training large ni models up to 2.5 times higher performance when running inference on them second is versatility in addition to running both training and inference TPU v5v offers eight different VM shapes to support the full breadth of llm and gen model sizes as large as 2 trillion parameters the divers VM shapes offer a wide range of computing horsepower and memory capacity which allows you to choose the optimal shape to suit your AI models and because it's a TPU it's been designed to scale using our largest multi-slice technology you can easily scale your model training across tens of thousands of chips Additionally you can conveniently leverage the exact same out of the box xlaa programming model to scale that you use today for all of you techn files out there I know you're craving to see the details TPU V5 is not the largest TPU we have seen but it is in fact the most efficient TPU to date in a world where we are confronted with a global EI chip shortage we have changed the way to think about serving our customers so we've chosen to make every chip count every TPU v5e chip packs a punch delivering nearly 400 ter Ops of computing while scaling 1,600 gbits per second over a blazing fast interchip interconnect each pod features 256 chips but it can be strung together across tens of thousands of chips using our Jupiter data center Network this elegant architecture allows us to offer v5e at an astonishing 54 cents per chip at the starting price starting today TPU v5v supercomputers are available for GA in North America with later availability coming to APAC and Amia regions to take the full advantage of TPO V5 platform we' have also invested in a complimentary software stack first off v5e supports both training and inference on AI leading Frameworks pych tensor flow and Jacks you can train and serve models end to end on tpus second we are introducing two new scaling Technologies multislice training enable Ables large scale training of AI models over tens of thousands of chips and multihost inference allows you to serve models as large as two trillion parameters last but not the very least Cloud tpus are now integrated with gke enabling seamless workloud orchestration furthermore integration with Vex AI will soon bring fully managed training and prediction workflows so this combination of elegant hardware and outof the-box software provides a coste efficient and productive endtoend AI development platform for you let's take a closer look at the inference stack we start with the trained pytorch jaxa tensorflow model we then export the model to a saved model using stable HL it's basically a portability layer between different AI Frameworks and compilers we run the saved model through the TPU entrance converter which performs initial optimization such as quantization we then compile this TPU compartible model using xlaa which does even more optimization including high level Fusion GMD sharding for data and model parallelm and lowlevel scheduling xcla also performs v5v Hardware specific optimizations and compilation so finally you resulted with a V5 optimized model that can run on tensor flow serving torch serve or saxs for Jack based models as you can see we're leveraging a robust software stack to take full advantage of the underlying powerful tpus we also bringing high throughput and low latency inference to Cloud TPU V5 when optimizing with our open xlaa compiler and the cloud TPU v5e delivers 2.5 times higher throughput per dollar for a range of state-of-the-art llm and ji models such as llama 2 gbd3 stable defusion 2.1 as well we can achieve this throughput performance without sacrificing latency in fact v5b achieves up to 1.7x speed up over the previous generation so what does all of this mean um it means that you can bring your most accurate most capable a models to realtime production applications at a fraction of what you would otherwise have to pay so llms and gen models continue to grow in size and the computational cost not just for training but also for serving also grows hand inand larges models require many many chips to fit and this is where flexibility of TPU V5 with fast interconnect up to 256 chip really shines Cloud TPU v5e supports INF frence on a variety of um model sizes a single v5e chip can run models up to 13 billion parameters and from there you can imagine the scale you can easily scale up to 256 chips and sometimes run models even with up to 2 trillion parameters and keep it still Within the Pod this is a quote from assembly AI one of our customers it's a testimonial from them for running their ASR models in production for their use cases well we saw the performance for inference use cases and the versatility of BM shapes that we offer let's take a closer look at training use cases with w5e we're able to achieve 2x higher training puff per dollar compared to the previous generations across lln 32 billion as well as gbd3 175 billion parameters we are also introducing aqt intake training we are able to see step function increase in perf per dollar for training workloads with multili technology we are able to scale Beyond power boundaries achieving near linear scaling for tens of thousands of chips in this specific use case we are able to scale a single workload to 16,000 chips with multi-slice training specifically customers can achieve near linear scaling across part boundaries to tens of thousands of chips with very few lines of code set up it's a full stack solution across gang scheduling orchestration compiler and runtime um as well as deep learning Frameworks to achieve this breathtaking scale character AI is a specific customer that has given us feedback on multi multi slice training it's truly a game changer for them and they are able to leverage through data center scale networking uh and they've used Jack's xlaa um to deliver out the box performance as well we saw inference and training performance graphs so far and here is an example of fine-tuning and this is specifically for Falcon llm we are able to achieve 1.9 times higher fine-tuning performance from the previous generations of tpus another great um customer testimonial from lightning AI they were able to achieve 1.9x higher puff per dollar for Pine tuning and again for a similar Falcon 7B llm compared to the previous generation of tpus again Cloud TPU V5 is now available globally uh we initially enabled access in Las Vegas um and also in the east east coast um there are many more locations opening up in North Americas soon to will'll also be opening up in Netherlands and Singapore this is our Global availability zones for cloud TP 5e we provide various consumption models to fit the needs of your business in use case you can spin up a machine on demand when you need them if you're able to but if you have a bursting workload or are testing any new workloads and On Demand uh is a perfect way to go once you understand your steady state requirements you can then purchase the committed use discounts in this case you're able to leverage up to 55% discounts across any region or a machine type family if you need additional savings then you can of course leverage spot VMS which have the lowest price for your most tolerant workloads shared reservations will also leverage discounts for on demand and um cuds our customers are able to leverage a mix or a wide variety of these consumption model to achieve optimal price for their business needs if you are interested um every customer will have a different path based on existing technology and business goals um our public website on cloud tpos has all the information about the system specs the scale and the documentation for getting started as well um you can also contact your sales representative to get immediate access to cloud tpus and and go from there with that said thank you so much for joining me today in my session there are wonderful other sessions that are being put together for women in ml Symposium I hope you have a wonderful rest of the day thank you so much for making [Music] it