Build high performance & cost-effective ML apps using Amazon SageMaker- AWS Virtual Workshop

Key Takeaways

foreign [Music] welcome to this session on Amazon sagemaker Fridays in today's session we're gonna discuss about the how to build high performance energy efficient and cost efficient applications using Amazon sagemaker supported by AWS strainium and inferentia instances my name is Arun Kumar lokanata I'm a machine learning Solutions architect part of Siege maker team I'll joined by two of my colleagues Raghu and Casey hey everyone this is Raghu ramesha I'm a senior machine learning architect in the sagemaker service team I help customers build deploy models on sagemaker hi everyone my name is Casey Tong I'm a senior solution architect Indiana Puna lab and our organizations builds custom silicon chips such as trainium and inferentia and my role is to help customers get onboarded with their model regardless of their platform of choice using trainium or inferentia glad to be here good let's get started so AI has been around for a very long time but when we see it's over the last decade or so we are seeing a flywheel a virtual cycle in evolving evolution of AIML so first it starts with innovation of a new model uh innovating ways to speed them up which typically unlocks a variety of use cases variety of applications and then companies when they build these applications they want to embed them into their products to Delight their customers which increases the demand and which in turn dries the need for additional invest investment to make a better infrastructure better tooling and better softwares this this of course internally fuels The Innovation and this cycle continues uh giving us a scope for evolving the AML landscape so let's review the latest trends in AI that are propelling the flywheel we looked in the previous slide so customers are driving a Innovation at an accelerated Pace uh to keep up with the applications use cases which meets their demand so possibly the artist Topic in today's deep learning Paradigm is large language models are the large models in general which are basically driven by Transformer based architectures so in recent years we have seen these models have gained popularity due to their state-of-the-art accuracy and scaling properties more customers are adopting these language models for their use cases and these models have scaled all the way from billions to billions to trillion parameters with we are seeing some customers or companies building trillion scale parameter models as well another critical enabler is the Advent of self-supervised learning which helps data scientists tap into the vast wealth of unstructured and unlabeled knowledge available online in turn this helps in boosting model accuracy and supporting training these large language models with billions and trillions of parameters the last element but really critical for Innovation is all the state of the art models such as the diffusions Falcon Lama and many of those are open source contributions this contributions helps increase adoptions for newer architectures and helping customers to kind of adopt them to their use cases to improve efficiencies so let's take a look at where because customers are using these models for various cases uh so large models they generally found usage in applications such as code generation where user large model to generate code recommendations or even do code reviews the Second Use case which is more popular is image generation you would have seen models like diffusion which can be used to kind of generate high quality images with a short snippet of text description um we also see applications of large models in music Generation video generation uh some of the models that generates videos based on a description uh when it comes to large language models we have seen a multitude of applications where customers are seeing success starting with text generation applications for productivity like say if you want to uh you know create a description for a product or any onboarding documents customers were able to use llms to generate text another use cases like chat or virtual assistant conversational interfaces where llms can be used with the context to answer questions pertaining to the customers and finally we also see adoptions in text summarizations where uh large language models are used for summarizing a long document uh right so with good quality outputs which are pretty much similar to what a human generates and finally there is a good application in such where llms are used to generate embeddings for documents which are indexed in databases or open search and retrieved based on the user query so there's a huge set of applications that are being driven by the large language models and why is this happening right so there is a a bunch of models that's been built and open source in the last few years uh we had bird like few years back which was Million parameter models and GPT came then there is llama there is blue uh we had Falcon so all these models are evolving at a very fast pace uh and uh each of these models when they are out they kind of achieve state of the art uh accuracy in in all the open tasks that's available and that way customers are also able to take this and easily adopt them or fine tune them to their use cases and most of cases they they can directly deploy these large models and then you know get uh immediate uh results for the need so let's take a look at some of the common challenges and trade-offs so first uh the information is developers are trading performance and cost so if you want to train this large models quickly uh you need to deploy them to a cluster of gpus this is expensive this is one of the main reasons uh building Foundation more llms are only for done by handful of people but more customers they want to generally adopt them uh and then use them for Downstream tasks the next thing developers are considering is uh how to you know use these models at a low cost uh so these models also two things right one is how do they reduce their cost at the same time how does that achieve the required performance like latency or throughput needs Etc uh the third challenge is how easy it's to use this uh large models uh how they can take this either fine tune them or deploy them based on their specific use cases lastly uh sustainability it's it's another important factor the bigger the model the more impact it has on the environment so how can we make sure we use the models which are right size for our use case um and uh so that we are also you know kind of considering its impact on the hand environment so to with with the above mentioned challenges into are considered into account uh we have been investing in coming up with most price performance energy efficient and uh scalable uh architectures are scalable Hardware that can be used to build this uh large models either tune them or deploy them using AWS built Hardware as you can see so we provide three options one is AWS inferentia which is which gives you low cost per inference in the cloud uh inferentia comes with two flavors we're gonna be discussing this in more detail uh later on in this webinar uh in fun and inferentia 2 and we also have a custom chipset for training which is AWS trainium all these chipsets uh they give you very good price per performance uh all the way from 40 to 70 percent lower cost also giving you a very good performance either in terms of latency during inferencing or in terms of training throughput during training so let's get started uh by looking at uh uh you know training first uh we will take a look at sagemaker uh like uh sagemaker isn't fully managed service that gives you multiple features or services for building training and deploying and monitoring your machine learning model uh sagemaker kind of uh gives ready to use features across your ml life cycle and let's start by looking at uh how do we train these models right so how do we train models are using sagemaker training uh by using uh Cranium instances so when you look at Large Scale training or a large model training uh there are three different aspects that we generally look at the first is the compute uh sagemaker provides compute uh with many options to choose from all the way from gpus to AWS trainium which is available as part of sagemaker we're going to focus on trainium today in today's session apart from compute uh we also need look at the platform features right so that helps in either fine tuning or you know training even training a foundation model with sagemaker so in terms of platform capabilities sagemaker allows you as to do large scale orchestration uh where you can spin up easily spin at the cluster with multiple nodes and Sage maker make sure you know it scales up and you know gives provides a cluster and it shuts down once your training is done uh we also make sure we do L checks uh on all the instances on the instances that we provide uh before uh you know you launch a training on stage maker so these L checks we do for uh GPU based instances and as part of sagemaker we provide something called as jumpstart where you can readily take any foundation model and you know you can deploy them or fine tune them depending on use case also to help a large model training and fine tuning we provide features like SSH integer training container uh memory profiling debugging and also options to load your data right so uh we might deal with large data uh when we deal with large models so we provide uh options to kind of stream your data from S3 or load data from a shared file system like an FSX onto your training cluster lastly uh once you have the platform in place then you need to have training libraries distributed training libraries and Frameworks on sagemaker we support tensorflow pytarch as one of the main supported Frameworks and we also have our own distributed trying libraries and you can also bring your own libraries we also support neuron distributed which runs which can be used for you know training large models on trainium foreign and before we dive deep into running a training on sagemaker using training so the titanium power trn1 instance uh it gives you 1.4 X the performance at off the cost uh com when compared to a similar uh instances on AWS so trinium comes with uh you know two two configurations one is TRN one two x large which has got one tranium Chips uh with two neurons part of a single chip and there is a trm132x large which comes with uh 16 trinium chips which is 32 neuron accelerators and each neuron accelerator comes with 16 GB of GPU so thereby making it 512 GB for a TRN 132x large and 32GB of instance memory uh in case of trm12x large so the network here is uh based on networking is based on second generation EFA EFA V2 which enables uh low latency and improved cluster level scaling so let's take a look uh at some of the supported models on train trn1 uh today uh so we support quite a lot of large language model architectures starting from GPT based models like GPT 2 GPT new X and we also support uh Lama llama2 you can train llama models 7B and 13B at this point in time and we also support uh support for Falcon and Bloom are coming soon in the near future yeah so when we talk in terms of performance uh we uh when we have Benchmark trainium against similar comparable accelerators on AWS uh where when we take a look at the birth model training uh we have we benchmarked all the way from one node uh to using 16 nodes on a cluster Cranium in terms of throughput as well as training costs when it comes to throughput trinium gives you 1.5 x uh more performance on a 16 node scale even on a single load scale you get a 1.2 x faster performance uh right and when it comes to cost trinium is 2.3 times cheaper when you are looking at a 16 node cluster scale and or even at a single load uh it's about one point at X lower cost when compared to a similar accelerator on AWS when it comes to llm training right so we have benchmarked against gpd3 uh GPT architecture with three different model configurations one is 24 billion parameter 46 billion parameter and 175 billion parameter uh on 24b parameter we see a 23 percent IR throughput uh when uh when we trained it using the trn132x large similarly uh we see a higher throughput when we use it on a 46b parameter model uh and 31 percent higher throughput then we train a 175 billion parameter model so with that background let's take a quick look on trinium architecture itself uh Cranium uh so each trinium chip uh like we saw trn12xlog has got uh one train in chip and then trn1 32x large as 16 training chips right so each strainium chips uh Scott uh two neuron cores uh it comes with two neuron cores and uh each of them has 16 GB of hbm memory and um it also comes with a separate engine for uh tensor engine which is used for larger Matrix operations such as your mat multiplication or convolutions uh there is also a separate scalar engine which can be used for element wise operations such as your activations and we also have a vector engine for aggregated operations such as like uh pooling vectorization Etc uh and it comes also with a GPS IMB engine which can be used for custom operations if you want to write your own custom operations uh we support that as well uh since we have a separate engine for each types of processing uh it makes a neuron much more efficient uh compared to other uh you know accelerators out there and in for inter neuron core Communications uh within an instance we have a neuron link V2 which enables fast Communications between each neuron course so uh with with that architecture we also have certain other features that are optimized for uh you know ever evolving large models out there so we have Noble data type support apart from the basic uh regular fp16bf16 uh Fe 32 kind of a support we also support uh cfp8 data types as well and much more and the second thing we have is uh an Hardware accelerator stochastic rounding so this is a probabilistic grounding mechanism that works at an Hardware level and this helps in improving the performance when you are training a model at Large Scale right and lastly we also provide abilities to write your custom operators we have a C plus plus wrapper mechanism through which you can write your own custom operators for neuron and what it takes for training on AWS right AWS training uh we support like mixer position format that you used with pytarch uh something like Tasha Auto cast which you can leverage uh to kind of train a model on training and for models itself we support two types of parallelism one is distributed in a parallel this is useful when your model fits in a single neuron core uh so uh and for data parallelism we support task or distributed and we have our own Collective Communications for all reduce all gather and ready scatter uh that works with task distributed similarly when dealing with large models we need a model sharding mechanism or a model parallelism mechanism uh we have support uh for like we have a neuron optimized neuron pneumomegatron and we also have a neuron distributed package uh which can be used to you know train large models like GPT newax or a llama 2 and so on all right so with that background on trinium uh let's take a look at uh what it takes to use a trinium instances with sagemaker training for running jobs on sagemaker training we provide a python SDK which can be installed as a pip packaged and which gives you easy mechanisms to submit a job onto a sagemaker training cluster so the SDK is much simpler to use you start by selecting uh which estimator you want we support Python's tensorflow we also have an again phase estimator if you are working with again quiz models and the pi torch estimator can also be leveraged for training arguing phase models as well ah few things that we provide uh to this estimator one is the entry point where you specify your training code and then you specify what instance type we need we can choose between TR and one two x large and 32x large we can also run a distributed training by providing the instance count and then we choose the framework question this is the pi torch version we are choosing 1.13 and python version to be 3.1 zero and once we specify that to run a distributed training a series maker estimator integrates directly with tortion so we enable that by providing the distributed mechanism for toss distributed enabled as true so what this internally does is like it uses touch Ram to launch a distributed training cluster and run the code that we specify on each of the neuron cores finally we pass the data to the estimator the data sits in S3 we pass it as a data Channel with sagemaker uses for training so behind the scenes when we submit uh the estimator.fet sagemaker is going to launch a cluster with trinium instance it's going to copy the training code from S3 a pairing code into that instance and it's going to copy the data from S3 into the training instance it also copies the neuron containers that we provide uh that can be used on trainum instances uh into the into the instance training uh and the training generally involves two phases when we use strenium the first thing is we compile the model uh so that it runs on neuron code since we have separate engines for tensor competition Pepto competition and so on so we need to do a pre compilation which automatically happens when you run the job on sagemaker it does the compilation first and then it runs the training poster compilation and once the training is done all the model artifact is copied to S3 so with that uh I will uh you know pass it on to my colleague Raghu who is going to run us through on inferencing using Sage maker and neuron based inferencia thanks sir uh this is great introduction on uh Siege maker training and how to use inference or how to use strainium on sagemaker so in this next section we look at how you can use uh the deployment aspect of it if you either you have trained your model on change maker or elsewhere how can sagemaker help you deploy and we'll also look at some aspects of cost saving and performance saving here and then Mike League will follow up with a session on inferential itself so uh looking at looking at this uh so we look at the section for deployment in sagemaker Amazon says maker is a managed service at accelerates every stage of machine learning life cycle that we discussed right so uh certain aspects of uh it can whatever you do on your training can be different from what you do on inference so sagemaker helps you provide an end user experience such as Jupiter notebooks a low code no code solution through sagemaker jumpstart and tools and libraries for you to easily deploy these models on uh sagemaker and also uh to top it off you have capabilities to monitor these models by using a sagemaker model Monitor and it all integrates well into your sagemaker pipelines where you can tie all things together have a logic where you can train and deploy your models monitor your models and then entire life cycle of management of these services like other function other AWS Services we also have an ability to provide a a access through apis and sdks as well as consoles and ideas so all the sales maker has different components that are natively built with each other you you can use these component inch independently as well so like I mentioned before you can either bring the model that is trained elsewhere or trained on sagemaker now specifically let's talk about the inference capabilities of sagemaker so if you look at this sagemaker offers wide range of inference options it provides you a managed resources and optimize inference a performance across your machine learning models uh to help you with that we provide you a way you can deploy it on a CPU instance GPU instance also on an inference here too which will concentrate on here uh in with an inferential to it provides you the highest performance of machine learning models it's a custom design chip with AWS specified for hosting models and EC to inferentia to instances uh based on inferential chips deliver up to uh 2.35 X throughput and up to 70 percent of the cost for inference and comparable to uh different GPU instances that are available for you and sagemaker also helps you support by providing multiple deployment options such as real-time influence serverless inference asynchronous inference and also batch influence for your offline workloads it uh in addition to that it also supports uh it also provides you uh cost saving options such as multi-model endpoints order scaling options that we'll touch upon in the next slides as well uh in addition to that um with salesmaker you have the flexibility to bring different model servers such as tensorflow uh touch serve Nvidia Triton MMS and so on and to help you in particular to deploy the large language models or generative AI models we have launched something called as the large model inference container which is purpose built containers or deep learning containers that we have to help you deploy these models at scale and get the lowest get the highest performance at lowest cost in addition we also support hugging face or TGI container which is text generation index containers as well uh so let's take a minute to look at what are the different dimensions that you can save costs and also increase your performance on sagemaker hosting so in this section we look at the software considerations uh what what affects your uh picking of your right instances and what are the different cost saving options such as asynchronous inference uh multi-modeling Points Auto scaling uh and also how your savings plan can help you save costs and uh and also towards the end we look at uh using influential to to speed up your inference and lower your cost so let's look at the first aspect of it so uh when you look at it the first first thing that we need to do is try to pick the right instance for that there are different kinds of considerations you might have to make so let's start with the software consideration so let's say your model uh model size is an important uh is important to understand how much memory that the model take so it's important to pick the right instance which has the enough GPU capacity that the model needs in if you're talking about the generative AI models some of the important considerations could be the output length input length uh the concurrency the search strategy such as the greedy search or the batch size Etc additionally uh another 20 percent of uh your uh memory is required for your attention cache or sharding the models across across memory and also your buffer memory so and one uh very useful way of getting a higher performance is by reducing the reducing the model size that is through model compression what this is basically is reducing the number of layers of connections or reducing the Precision of Weights uh and activations by using quantization methods and we'll focus on uh quantization which involves reducing the Precision of Weights let's say from fp32 that requires four bytes per uh for using to like say fp16 which only requires two bytes by doing that you are almost reducing the size of the model by half and that helps you uh keep more copies of that model in memory or pack more models into into the memory of the same instance so with uh in addition we can use like flash attention which is a new algorithm that uh that uh computers extract attention from Far few memory accesses our goal here is to avoid reading and writing the attention metrics in and out of the memory and it also reduces the overall inference time so it basically uh think of it this way as it reduces the memory needs and writes between the GPU for high uh high bandwidth memory or hbm and the gpus on on chip so in addition you have different optimization strategies such as you can optimize with Frameworks such as deep speed faster transform accelerate to reduce your latency and also enclose your throughput resulting in lowest cost per day uh token that that your model requires uh some of the some of the questions uh you'll need to ask before choosing the right hardware for your instances the first uh and the foremost choice would usually be if the model is compatible within friendship too if so it will give you the best uh or the highest uh price per performance you can check the models that are supported on our uh in Friendship page if that fits into the bill that should be your first option if not we could look at like some something like a G5 instance which is a single GPU instance like a uh like a g5.2x large if if it that fits in if your model fits into a single uh single GPU instance like this uh that's that's the next best option if not you can look if you have a bigger model you can look at like the bigger instances of the G5 family such as like 12x large or 48x large next uh you would use a G4 DN instances uh if your models are smaller in size and this usually comes with a larger latencies so if your application is uh is capable of hosting a larger model this might not be a good fit for you and finally if you have a p4d instance which offers you the lowest uh lowest latency and accommodates the largest largest of models uh bigger than G5 instances but uh these are also one of the most expensive instances that we have so so be sure to maximize the GPU utilization of that instance pack more than one model for instance or more more copies of that model per instance to maximize the gpus here uh in addition most of the Transformers are trained with mixed Precision either on fp16 or fp32 or BF BF uh 16. so this cuts down the amount of memory required to train models and also it uh the memory required to run the inference so we can cache these models from FP 32 to fp16 or even going down to intake which reduces even the less memory that is required without substantially taking a performance hit so these are some of the considerations you would choose when uh you take into consideration before choosing the right instance and to help you with that we have the next capability so in order for you to help benchmark and load test models in production and also to speed up your model testing process from a POC to a production State we have introduced a new capability called sagemaker inference recommender uh with using the sagemaker inference recommender you you'll spend lot less time and also will provide you the set of the various set of parameters that are required to uh and the container configurations and environment variables suggestions of that so that you can array with the best a best instance and also the configuration that will work best for your workload in fact most of our custom most of our customers have used inference recommended to right size their endpoint and saved up to 30 percent of improvement in price book of price per performance foreign saving method is um and is something called as Sage maker asynchronous inference this is uh this typically useful when you have uh pay when you have workloads which are intermittent in nature let's say you have a large video stream or um or textual data like an NLP uh problem that there but you don't have um you don't have a stream that's coming in all the time so you don't want to have an endpoint that's always up and running so this is where sagemaker in async conference can recommend async inference can be very useful uh when you don't have this traffic coming in you would scale down the instance to zero and then only scale up when it's required so it's a cost saving aspect as well and it works very similar to your offline inference where you can put in your inputs into an S3 bucket we'll read from that S3 bucket and then we have a managed queue here which basically cues the amount of request that you can send uh all the number of requests that you can you get onto the end point and then you can control how much of uh the uh the request you can you can also decide how many instances that you want behind the end point and appropriately choose how many requests you want to pull out of the queue so this is a very useful mechanism and you have intermittent traffic and when you want to save costs additionally uh as part of your real-time inference which is your always on end points like I mentioned you uh uh and useful capabilities called uh sagemaker multi-model endpoints uh with uh sagemaker multi-modeling points it provides a scalable and cost effective way of deploying large number of machine mod machine learning models up to thousands of them it gives you an ability to behind the same endpoint it provides you an ability to deploy these models behind a same endpoint and uh this will be particularly useful when you have smaller models and which are of very similar Frameworks um and you can potentially save uh and you can you can potentially save up to 90 of your cost by using multi-modeling points this is by uh By Design when whenever you you have a model that's loaded on a set of models behind that endpoint we will dynamically load that model and unload that models behind it endpoint as and when you make invocations and we will be sure to keep the models in memory which are being accessed more often so that you don't have any cold start times there well finally what we have is uh uh using this uh using sagemaker uh uh using order scaling policies with uh Auto scaling policies uh you can dynamically adjust the number of instances provisioned behind the endpoint as your workload changes uh there are multiple uh Target tracking policies so um scheduled scaling policies multiple on-demand policies so basically whenever your workload increases Auto scaling will bring in more instances and make sure you have more availability and also whenever your workload decreases like if you're using a nasync endpoint per se you can scale down the instances to zero or if you're using a real-time inference you can scale down the instance to one and remove the unnecessary instances and don't pay for the over uh provision instances that you aren't using so uh this is a very good cost saving aspect of it that uh that most of our customers leverage and again benefit in finally uh we have the sagemaker Savings Plan sagemaker savings plan is a flexible pricing model for sagemaker which which in exchange of uh for a commitment of uh consistent usage of amount or measured in dollar per hour for one or two uh one or three year term sales maker plans provides the most flexible way to help you reduce cost by almost 64 percent this does it by automatically uh using the eligible instances such as notebook instances on-demand or instances Sage maker processing influence and batching like real-time uh batching batch instances regardless of the family or the size of the region of that instance uh that kind of brings it to the end um I want to hand it off we'll start again uh with that I want to hand off uh now that you've looked at different cost saving options which says Michael let's look at one aspect of it where we'll look at uh how stage maker inferentia chips help you gain more performance and lower costs for influence with that I'd want to hand off to Casey all right thank you rabu I'm gonna start talking about uh imp2 or inferential 2 instances and how you should be able to leverage different size of instances for different purposes and as I stated here we designed in two instance or rather influential two chips as a second generation coming off the inferential one which is the first generation Uh custom silicon chip built by AWS exclusively to run inferencing workload in AWS cloud and ever since then we see the need to evolve in terms of a custom silicon design in terms of the kind of a model that we can serve the customer because as we know the models are getting larger and larger and the internal structures are getting more and more complicated so with that in mind we design and we Custom Design and build in two instances as well as the the chip behind it that's powering it known as inferential two so for the in two comparing to the in one the first generation of the AWS inferential chip uh we see up to four times of higher throughput and also 10 times lower latencies when it runs the same model comparing the two so and also there is also a bump in the accelerator memory bandwidth as well and this becomes really important when you talk about uh how do you inference how do you run inference job for large language model for example or for generative AI models such as a stable diffusion that translates text into image for example these models are complicated and demanding in terms of memory bandwidth in their own rights so let's look at the four different tiers of instant sizes that we offer okay uh the for the first two tier in the instant size you will notice that there is one inferential two chip and one inferential two chip consists of two neuron cores so you can think of it as two gpus inside of a one inferential two chips and and Then followed by the next two tier and you have either six or two up to twelve inferential chips and so that will give you up to 24 neuron cores or equivalent of 24 gpus running separate workers and as you can see that for the first two tier uh these are single chip instances so that there's no need to con to connect the chip to chip with a neuron link however if you as you go up to the larger instance size the workload becomes important enough especially for large language models that you have to Shard the model across multiple chips multiple differential two chips then how do you make sure that they communicate how do you make sure one part of the model flows to the next well that lies in the way the neuron how the neural links works so that becomes very important when you talk about inferencing a large language model and produce the output in a timely manner so for that we designed the neuron link which is a direct communication between instances and minimized data movement and bypass the data going through CPU so that also reduce a lot in terms of the latency between the data movement and as you can see with the instance size increase we also increase in terms of corresponding CPU as well as as well as the uh the instance networking speed so this is all become this is all very important for large language model inferencing in terms of the speed in terms of the throughput now I'm going to talk about AWS inferential tools chip and its internal cores and on this diagram as you can see it's actually very very similar to trainium uh in fact that is exactly what I want to touch upon AWS influential 2 and trainium actually shares over 90 percent of similarity in terms of this at the Silicon level and again what you're seeing is that you you see that within the influential 2 chip there are two neuron cores again think of each neuron core as a separate GPU and within that particular neuron core this is where things starts to be a little bit interesting you notice that we separated out into three different engines there's a tensor engine there's a vector engine and there's scalar engine and this is exactly like how uh previously uh a room explained that tensor engine handles large matrix multiplication job Vector engine handles aggregation and the scalar engine handles uh operations such as activation function and then in addition to that uh there's also a collective communication engine that you see on the top right and the this is actually built to handle fuse Ops as well as a gradient accumulation and gradient compression uh during the um during the during the large language model influencing and to reduce the data movement between uh the code between a neuron core as well as from the neuron core to the CPU and the result of it is that fuse Ops for data movement can be run in parallel at the same time the compute engine is doing its job the tensor engine is doing the matrix multiplication and scalar engines doing the activation and so forth so we parallelize as much as we can from operation standpoint and also uh noticeably we also have the neuron link V2 uh which like I mentioned uh is responsible for direct data movement between neuron device between inflation to chip to the next information to Chip and this becomes very important when it comes to large language model inferencing because the model is so large that you have to Shard the model and spread it across multiple inferential chips even during one inferencing round you need to you need to kind of collaborate if you will between the chips in terms of how data is being moved from one level of the model to the next and this becomes important that only if you have a fast speed linkage between the chip that can this be done so that's what neural link V2 is good for and in addition we also have the network connectivity up to 100 gigabit per second in the larger in the largest instance in 2.48x large and also the high bandwidth memory is used to hold and cache if you will as much model weights as you can uh so that the model weights doesn't have to be recalled and load and unload uh during the inferencing step so all these things add up to reducing latency and also increasing a throughput AWS neuron SDK makes it easier for developer with AWS trainium and AWS inferential access so with the powerful Hardware that's custom built and purposely built for large language model inferencing job in order for user to access it we need to also have a software Stacks that can just that can make these Hardware features accessible and to that end we make sure that our framework support is consistent with what's most popular and the most advanced in the industry in the community so for example we see pytorch and tensorflow as the main framework that we provide support ongoing support and evolution of neuron SDK in order to fit into the pytorch ecosystem as well as tensorflow ecosystem and also we have strategic partnership with hugging face who is the large probably the most famous and one of the largest open source model repo in the industry and also we are also part of the open xla Consortium for even larger and more complicated model built out in the future now for neuron package itself we provide compiler runtime and the developer as well as monitoring tools such as neuron top for user to take a look as well as to leverage the hardware features um that the inferential 2 supports and on the bottom and you see that these are the links to our official documentation for neuron as well as a GitHub repo for a lot of example and code base for you to try it out so long story short AWS neuron is the SDK built exclusively for you to leverage and access Hardware uh that we design such as training and inferentia now I'm going to talk about popular model support for trn1 and M2 and this is as of the latest update that we have and at the recording time currently as you can see we're up at the neuron 2.13 release and on these two tables tables on the left is about language model and table on the right is about computer vision model so as you can imagine in a generative AI field invariably it comes down to these two domains generating language or generating image or video in the future right so um you will notice that in both cases we support inferencing more uh than which we do support training and that is because at a lot of our customers want to start with a model that they already have access to they're happy with their model and they are ready to deploy it in production so or that they train their model elsewhere from some other domain or from some other infrastructure and they have the model weights they have the model file and now they want to put it in production so naturally uh we look at the customer priority and it turns out that inference is is a much higher demand a much urgent demand from the customer perspective because this tells them that AWS is ready to support the model that they're building house or the open source model that it grabbed from the info the ecosystem and they they are ready to deploy it and now they're looking for a cost effective solution and AWS offers it so that's why we focus on inferencing First and as you can see on the left from the language model perspective uh a year ago we started off and making sure that we support Bert uh as well as of different flavors of Bert and today we're very very solid and that we support almost all the bird models areas available in the industry and this year we started to work on uh decoder-based models such as the GPT type of the model and again you see that our coverage is starting to emerge in that GPT model families we start up with inferencing and then you're going to start to see more and more coming soon for example like gbdj uh like Bloom for for for training for model training support and just very recently we announced the support for llama based model for both inferencing as well as support and also for the encoder based model encoder decoder model such as T5 we have training support as well and the inferencing support is coming so and for computer vision models such as diffusion uh stable diffusion model currently we support three different types of a stable diffusion model for inferencing and then also something smaller Vision Transformers yes uh which we have a very good coverage for the both training and inferencing as well next slide is uh in two Benchmark for large language model inferencing and from here you can see that um our um in two instance actually provides a very good throughput for example for some of these large language models that's currently in the in Trend and starting with the Amazon's own Titan model to some of the public Foundation model and then also the third-party models next slide is about the Benchmark result that we see through stage maker and AWS internship 2. and in here we put together a side-by-side comparison of of the token per second or rather throughput if you will of a gpd4j model and as you can see in the comparison with a competitive GPU instances our throughput is actually much higher and likewise with latency when it comes to generating image for the computer vision model such as stable diffusion we provide we provide exactly the same latency in terms of in terms of image generation uh when it compares to a competitive GPU instance but at the fraction of cost now I'm going to talk about something very unique and specific uh to the runtime architecture uh that it's known as how do you suppose how do you actually support large language model in inferencing and how do you make it work within a sagemaker ecosystem and as it turned out we make sure that our Transformers neural X package which is which is a inferencing package that anapuna team build to support large language model inferencing is we make sure that it's compatible with the sagemaker architecture and in such a way that it is wrapped around in a djl serving and from the djl serving it's exposed through sagemaker user interface but let's take a look at the inner working of what does it take to actually serve a large language model it turns out that large language models such as GPT level kind of a model is so big that you cannot just Shard it a long layer by layer and then split it across neuron core or neuron devices you have to split horizontally meaning that within one layer you have to split the tensor weights if you will across it in such a way that the tensor weights are distributed itself as a tensor it's split among all different neuron cores so that's why transforming neuron X does and that's what the parameters such as tensor parallel actually means it means that I'm gonna this is how I'm gonna split the model weights basically not from layer to layer but rather cross a layer and horizontally through the tensor and the way to do it is using the Transformers neuron X package that as you can see that if the tensor parallel degree is four that means that I'm going to Shard the model from four paths to the front forward pass all the way to the output horizontally in four parts and Transformer neural X will be responsible for collecting um the model Waits forward propagation aggregated together and the producing output and now you can see the importance of the neuron link V2 and without neural link B2 this wouldn't be possible because you have to link together uh the forward propagation result of one part of the tensor to the next in a speedy manner otherwise a user experience it's not going to be ideal because it would take you forever to wait for the next worker to propagate the likewise the forward pass of one portion of the tensor all the way through and this is why we designed the neural link V2 for exactly this purpose and now um reaching the towards the end of the presentation I'd like to point out these two useful uh website uh one of which is a result of our collaboration with hugging face Optimum team and the hugging phase team built out an Optimum neuron API to really enable a hugging face model users to use one common or unified API style to run training and to run inferencing job and on the bottom there's a link to AWS neuron link for the documentation and in there they contain a lot of examples as well as introductory materials to a lot of things that we talked about today for example tensor parallelism why do you need it and for large language models how do you use it that as a knowledge to select the instant size for your inferencing job so with that regard um today I think what we have covered is a distributed training with AWS trainium uh and then we also discussed the neuron highlights as well as a training and inferencing examples shown um in a sagemaker and then also um with trinium and inference instance details and the kind of a benchmark you can expect from us and next these are just a few customer studies so for Finch Computing which is one of our customers that leverages M2 and trn1 for their NLP job load and this is their testimony so with that I'm going to end right here thank you very much for your attention [Music] thank you foreign [Music]