Distributed ML training using PyTorch and Amazon Elastic Kubernetes Service - AWS Virtual Workshop

Key Takeaways

[Music] hello everyone and welcome to distributed model training workshop on aws eks my name is alex jankowski i'm a principal solutions architect in the ml frameworks team at aws and i specialize in orchestration of machine learning workloads on ws using containers hi everyone i'm shubha kombatkone i'm a specialist in the frameworks team focused on helping customers build their ml models on ec2 and open source tools and technologies yeah hi everyone my name is imran yunus and i'm a principal solutions architect at ml frameworks team and i focus on distributed machine learning and deep learning kind of problems in today's workshop we will follow a simple workflow first we'll set up our development environment using cloud9 on aws then we'll create an eks cluster we'll create a shared volume where we'll we'll store the data later we will build a deep learning container that has our distributed training code will execute a job to download the training data and store it on the shared volume then we will run distributed model training using phy torch and torch elastic and finally we'll run a test to see how good of a model we have built in the workshop we will use the following resources the event box link provided at the top will let the participants provision a temporary aws account where they can complete all of the steps in the workshop we will use a github open source repository on the second link which will allow um everyone to follow through the workshop steps and copy and paste code from the repository into their development environment to complete all of the steps in the process first let's log on to the aws console and set up our ide i will be switching screens between this presentation which describes our steps and my demo screen here as well as the open source repository full view so to start off i will go to the demo launch page which can be accessed through the short link bitly forward slash distributed training workshop hyphen eks event this will provide a launch button for us and let me make the font bigger here when you click the launch button you will be provisioned and assigned a temporary aws account you'll have to agree to the terms and conditions for the workshop and on the next screen you'll have a chance to enter your email address so that you can be sent a temporary password to log on to the aws console while i'm waiting for the passcode to arrive in my mail let me scroll to the distributed model training workshop repository and describe briefly the setup that we're going to use in today's lab so we'll execute everything inside your browser we're not going to need to all of the steps will run on the cloud we're going to work inside the cloud 9 ide which is a full development environment running on ec2 instances in aws and from here we will clone the github repository and execute all of the scripts for all the steps in the workshop they'll create infrastructure inside the vpc kubernetes or managed eks cluster with a onenote group where our workloads will run and we will build the container here push it to the elastic container registry and then we will instruct eks to run that container in several different configurations to do the distributed training the nodes in the cluster will pull the container from the container registry and run the job here while the task while the data is being stored in an elastic file system which is mounted and shared to all of the nodes and pods in the cluster so i have now received my temporary password code and i'll paste it here to gain access to my console once you're authenticated you will be presented with the team dashboard screen if you click the aws console button you will also see temporary credentials on the screen it is important to copy those and save them locally on in a temporary file because we will need to refer to these credentials later in the workshop once you've copied the credentials click the o console button and that will log you in to the aws console provide access to your temporary account as i mentioned we are going to use the cloud9 ide which has already been set up in your account so we will just type cloud9 and when the cloud9 service an ide called workshop is already provisioned in the account so i'll just click the open ide button and this will bring us to our environment where we will run all of the steps from the workshop the cloud9 ide is as i mentioned um fully functional development environment which runs on aws ec2 when not in use it will actually stop the ec2 instance so you're not accumulating any charges and the first time you want to use it after a period of inactivity if the instance is shut down it will automatically start up and resume the state that you had when you last were active on the environment so i'll use two screens here just so i can show the um the steps better however in your workshop it is sufficient to just use one of the screens i'm pasting the crystals that are saved from the earlier screen in in both of my terminal windows here um but i will use just the bottom window for the time being to create the eks cluster so uh the first this workshop can be uh completed um in your own aws account as well and if you wanted to do that instead of using a temporary account provided by aws then you could follow the instructions in the setup document in the prerequisite section so this document takes you through setting up your own cloud9 id and giving permissions to your aws account and configuring the cloud9 ide so that it can execute all of the steps in the workshop and as a last step it shows how to clone the open source repository that we'll be using today so to clone the repository you could also just copy the clone link from the repository itself and execute git clone which provides a little copy of burst code the repository is structured in the various steps we'll go through step one through six here and each of the steps is executed in its own directory first we will go into the step to create an eks cluster for that we switch the networking directory to one create cluster and we'll execute the scripts provided in this directory in order first we'll provide credentials to the local cloud9 terminal by pasting the aws access key id and aws secret access key that we have saved we use us west 2 as a region and output format we are going to use is json as soon as we um complete this configuration uh the cloud9 ide asks us a couple of questions so the first uh one we would say cancel do not use managed temporary credentials and then permanently disable to basically opt for using the credentials that we just entered from the ide for the workshop the second step would help us install some tools in the this environment like eksctl cubectl and cubectx which will help us throughout the workshop to provision the infrastructure and to interact with our kubernetes cluster so that we can run distributed model training on it and in order for these changes to take effect in your shell it is not required but helpful to execute this command source home folder dot bash rc this basically creates some aliases in your environment that you can use as shortcuts instead of typing cubectl later you'll be able to just type k to do the same thing i'll do i'll source that bash resource file in my other shell as well so i can as the shortcuts from either window and now we're ready to run the script create clutter which will actually start the process of creating a kubernetes cluster um just to explain a little bit about kubernetes a kubernetes cluster is a group of virtual machines running on any given infrastructure it could be running on premises around vpc or um as a managed service on aws the elastic kubernetes service is uh basically the open source kubernetes distribution which runs on aws in a managed way which means that aws takes responsibility for provisioning and managing the cluster while you are able to have full control of the jobs and workloads that are executed on the cluster itself when you're using eks the master node that contains the api server runs on aws owned infrastructure and the nodes of the cluster which are the node group we're going to use to run in your own vpc um so you're able to see the nodes where your pods will run but you are not able to have um shell access into the api server which is managed for you by the elastic kubernetes service so to create the cluster we uh basically use this cluster manifest which says that we're going to create a cluster called do eks of this version in this region we're going to use two availability zones and integrate it with um i am with through openid connect i will have one managed group called workshop node which is of type c5 for x large and nodes from this node group will be distributed between these two availability zones for high availability um the ksctl command is what actually kicks off creating the cluster it shows it uses this uh manifest that we just reviewed and this is the command with one command you basically kick off the cluster creation what this does in the background is generate cloud formation scripts for you and provision all of the infrastructure necessary in order to get your cluster up and running i will quickly show the cloud formation um tasks that are going on in the background uh this is for informational purposes only we can see that there's this uh do eks cluster stack that uh is being created and there are several events uh that are in progress so um with this while the cluster is being created and we're waiting for it to become ready let's uh pass the microphone to my colleague shubha kumbadakoni to give us an overview of aws's ml capabilities and the reasons why we want to do distributed model training on aws hello everyone um thanks alex uh hello everyone and welcome to the workshop on distributed training using eks uh as you wait for the clusters to um be set up i wanted to take some time today to kind of talk about why we are seeing distributed training becoming increasingly relevant um and what what are some of the challenges that you need to be considering from an infrastructure perspective as you set up your distributed training architectures what we've seen in the aiml landscape is that models are increasingly becoming larger and more complex and there are a couple of reasons why that's happening more customers are adopting deep learning because the amount of data that they need to analyze is increasing um it's becoming pretty large they're generating terabytes of data on a daily basis and they want to be able to train their models on these large amounts of data so they are increasingly adopting deep learning which relies on neural networks as well as transformer based models and these models are fairly large and by large we mean they have a large number of parameters in the order of millions and much much more and if you look at how the the sizes of these models are increasing you're aware you see that there is an exponential increase in of the complexity if you think about a birth large in 2018 to a gpd3 which has 175 billion parameter parameters to now models that are even you know a trillion parameters large the size of models is increasing exponentially but the underlying compute that is required for training these models is not really growing at 10x speeds so that introduces a challenge of how do you then train these models uh in a time that is not prohibitively too large how do you make sure that your training now doesn't take months and years and how do you effectively accelerate your training that's where distributed training really comes into the picture and is important and what do we mean by distributed training it's really um the term that is used when a single training task is no longer relegated to just a single gpu or a single server but instead is spread across multiple gpus and how you architect it is what we're essentially covering in our workshop series there are a couple of challenges that are inherent to distributed training uh you need hyper-scale compute you need to be able to easily deploy large clusters sometimes at hpc scales for being uh for for training these large models and the cloud if you think about it is ideally um is ideal for that kind of hyperscale compute because you can get compute on demand and you don't have to spend months setting up your on-premises infrastructure for doing these uh training jobs training is also done typically on a large number of multi-gpu nodes so you need uh more powerful servers that uh that can deliver the performance that you need the nodes also need to work in a distributed fa in a parallel fashion or a distributive fashion to complete the training job so there is um high speed internode communication that becomes necessary as well as gpu to gpu communication needs to be pretty fast data has to be shared across all the nodes so there needs to be a distributed file system that needs to be uh present and as you now think about scaling your training across multiple nodes fault tolerance becomes important so your training job needs to be resumable if you get interrupted you need to be able to pick up from where you left and not lose all of the the work that has been done if you think about distributed training there are certain constructs that are in place today data parallelism is um an intuitive way in which all of the nodes have the model but they work on a subset of the data of the training data and then there's model parallelism where the model is sharded across all of the nodes but every node or every server works on all of the training data set and in model parallelism we do have pipeline parallelism and tensor parallelism and each of these have their own advantages and they come with their own challenges as well so let's talk about data parallelism so like i said this is a fairly intuitive um distributed training construct in this each of the worker nodes have um can store the entire model in their memory and the model works on a subset of the training data which is known as a mini batch so what happens is every worker node trains on a subset of the data and once um the model has converged on that subset which is called a mini batch the new model parameters or what we call the gradients are communicated across all of the worker nodes so that at the end of that communication phase all the worker nodes in your cluster have the updated model parameters now once this is done the next batch of trade uh training data is uh sharded across all the four or all the worker nodes in your cluster and the process repeats till we reach the required uh accuracy until the model converges so that is data parallelism but model parallelism is when the model is sharded across all of the nodes in your cluster and this can happen because the model is too large to fit into the memory of a single node anymore so in that case you need to look at ways in which you can shard the model across all of the workers and then train it on the entire data set that is available for instance in the pipeline parallelism uh what happens is that every layer of your model is assigned to a specific worker so we form a pipeline of execution in this particular example we see that layers 1 and 2 are assigned to worker 0. layer 3 is assigned to worker 1 and layer 4 is assigned to worker node 2. they so that becomes your pipeline of execution each worker node will execute or will train on the entire training data set while pipeline parallelism is a good way to um address the problem when the model becomes really large to fit in onto a single node um the pipeline parallelism uh has an inherent challenge where one one particular node or one particular um stage of the pipeline can block the uh can become a blocker for the execution because it is much slower than the remaining stages tensor parallelism is called we can also call it intra layer model parallelism so this is another approach of a model parallelism where each worker node has every layer of the model the um though that way you're kind of overcoming the inherent challenge of pipeline parallelism but what tensor parallelism requires is a high degree of synchronization between all of the workers um and ensuring that all of the updated model gradients and parameters are in lockstep across all of the nodes in your cluster now aws has the broadest and most complete set of machine learning services that can help meet you way wherever you are based on the kind of flexibility and the kind of control that you want over your machine learning applications if customers want to do api calls and that's the kind of flexibility and that's the kind of low lift they want we do have ai services like recognition poly and comprehend that can provide that if you want to build train and deploy your models but you want to offload the infrastructure selection the ci cd process um of maintaining your model then amazon sage maker is a really powerful tool to help enable that and we in fact have a workshop next week where we go over how to do distributed training using amazon sagemaker and the ddp libraries there but if you want to have a high level of autonomy with the tools that you use with the infrastructure options that you use with the open source technologies that you want to use then we do have powerful infrastructure services a wide range of orchestration tools and software that you can leverage to build your distributed training um as well as inference architectures double clicking on that point we have a pretty broad range of high performance accelerated compute instances that range from a100 powered uh instances like p4d uh g5 which has the a10g uh gpus dl1 which has the intel havana gaudi accelerators and trainium which is powered by uh aws silicon and trn1 instances that are currently in preview are powered by trainium and it's set up for distributed training with a high-speed networking and powerful accelerators so we have a choice of compute instances that you can choose based upon your price that is the cost as well as the performance that you need for your training jobs we do have storage services like ef efs fsx for luster s3 and uh efa which provides high speed networking it's a networking service called elastic fabric adapter that supports up to 400 gbps of high-speed networking we of course have orchestration services that can both be managed like eks and ecs as well as open source software that you can leverage like kubeflow airflow and build your architectures on aws i wanted to close by talking about our customer ai 21 labs who is a startup that's based out of israel that has um a product called word tune that is a writing assistant and they basically use an nlp model they use a jurassic one jumbo model which has 178 billion parameters and we helped them build out the distributed training architecture on p4d ultra clusters using the high speed networking of efa and they were able to deploy their distributed training architecture and train their model on aws much more faster and scale much more easily when compared to um doing this on premises so the cloud or aws def with hyper scale compute readily available powerful infrastructure services readily available can really accelerate the time to deploy the training architectures as well as meet the time to train requirements that you may have with that i want to hand it over to imran to go over the code specifics of what we're doing today uh thank you shubha so now i'm going to basically explain what task we're going to run on our egas cluster so we are going to train a deep learning model on an image data the data set that we are using is a c410 data set it's publicly available it consists of 6000 images which are categorized in 10 classes and each image is 32 by 32 pixels and we're gonna train a deep convolution neural network the code is provided in the workshop uh the convolutional neural network is divided it has a six combination layers in divided in three groups each group has two convolution layers followed by a max pooling layer and a batch norm layer the network ends in a linear layer and then at the end i have a final layer output layer with the 10 outputs because i have 10 classes in the network the model is wrapped in a class which i just called my cnn class so in the training script which is also provided in your lab we have a you know the very traditional pie torch model training code with the two differences and the other minus difference value also the main thing is that we are importing a data distributed parallel class and we're also loading a elastic distributed sampler so i will show you how these are used uh in the top i have some parameters that we can set in the yaml file when you launch the job for example the batch norm so it's the batch size learning date etc etc the next thing is the data loader so whenever you train a python model you have to first define a data loader in our case we are using the cipher 10 data and the cipher 10 data when you download it it has a tar gz file when you untie it what you get is a bunch of pixel files each pickle file has basically data stored in a list of arrays numpy arrays they are not stored as images so when i unpickle with this small utility which is also provided in the lab what i get back is a images list of arrays and the label which is also numpyray so before moving forward we have to convert these numpy arrays into a tensor dataset which is the pytorch class so this is what the get tensor data set utility does again it's also in the in the code um and then i have to create my data loader for this specific work we have to basically provide a data sampler to the data loader and that is going to be elastic distributed data sampler which is very specific to the training on eks clusters once i have my data loader then i have to create my model so the model as i mentioned earlier is my cnn class so i'm creating my model but i have to then wrap it into a distributed data parallel class and that's all you need to do when you're distributing model across multiple gpus so distributed data parallel basically takes care of copying the model on every gpu available the rest of the stuff is pretty standard pi torch i have to create my loss function and i have to create my optimizer then in the main function i have to start from defining which backend we are going to use for uh gradient averaging and in because we are running on cpu we are going to use the glue backhand uh if you are running on gpus you have to use knuckle backhand the other thing i need to get is the rank because in eks when you create a cluster every node is every part in this training loop will have a rank assigned to it which is unique integer id and you want to get that because that is also used to define where which part is going to save your data or last model and then i'm basically just you know calling this function that i defined earlier getting my chain data loader and i'm initializing my model i also want to save and load the checkpoints so this function here will see if there's a checkpoint already stored and it will load that checkpoint and start from that if no checkpoint is there it will start from the very beginning this is needed if your training gets interrupted then you basically have to go back and run them uh again and and then it will start from the previously stored checkpoint finally i have the main training loop and as you can see it's basically pretty standard my first training loop there's not much difference here so i'm basically looping over all the epochs and then i'm looking over my data loader uh it returns me a batch of input and labels uh i'm doing a forward pass i'm calculating my loss i'm doing a backward pass and i'm stepping forward and finally i'm just writing down some parameters like the loss basically for every iteration just to see how it is progressing one thing that is different is that storing the checkpoint so for each epoch i'm storing my checkpoint and the checkpoint basically stores the model weights at that time and the what epoch is there and the optimizer state and i'm storing it only on one part if you are running multiple paths you don't want to write the model from every part you only want to write the model from one part and that's why i'm using the right once all the epochs are done uh we're going to store the final model and that's it so that's the whole code for model training once the model is training is finished we're going to basically run the test code to see what is the final accuracy for that i have another script here again a pretty standard python this script will not use torch elastic this will run only on one part because we are just doing a forward pass with the model so i'm only going to load the data loader again i'm loading the c far test data now which has only one pickle file and i have to convert into a tensor data set because i get the numpy arrays back from the pickle file and then i have to create a data loader this time i'm not providing any data sampler so it's going to run on a single node just like a single machine and the main function starts from the calling this function which gets the data loader and then i'm loading the model from the stored model and finally i just have to basically do a forward pass on this model using the test data and i'm going to calculate some of the metrics and that's it that's the whole code uh i do want to point out that in all these cases the all the data is being read from a common uh disk which is mounted on every node so whenever i say that i'm saving a mod and loading model this is happening on a common disk that is mounted to evernote uh with that i think i'm gonna basically return to alex to continue thank you imran uh and now let's go back to our development environment uh by now the cluster has completed uh creating as we can see uh the script uh is done creating the cluster and the local environment was updated with the new cluster context which is saved in the local cube config and we also were able to execute a command to see the available nodes in the cluster which are displayed here so from this point i am going to split my screen into here um if i execute the command to get the nodes i can do that at the at the same time at the top of the screen i will watch the all of the parts that are running in the cluster here so that we can keep an eye on this going on so a little spelling mistake here so um as we can see nothing is really deployed in the cluster yet since we just created it what we're going to do is execute script 1 4 to deploy packages and this deploys the kubernetes metrics server the torch elastic job controller and the lcd server to the cluster the metric server we use to monitor node utilization during the training jobs the elastic job controller is what contains the custom resource definition for elastic jobs which we use for distributed training and the fcd instance is a coordinator for the workers in the distributor in the distributed training job which allow them to coordinate the work between themselves so this completes the server the cluster setup portion of the workshop next we're going to create a shared volume which can be mounted in each of the pods that you're going to run in the cluster i'll scroll here to remind us that we are now at step two to create a shared volume uh to do that we will change our current directory to the create volume step and we have two scripts to execute here uh first we will use the aws cli to provision an efs volume in the region that we have configured and in our current vpc so this creates the efs volume we can go to the console type efs and we'll be able to see that volume created in our account and in order for this volume to be able to be mounted by each of the nodes in each of the availability zones uh for each subnet that belongs to these availability zones we're going to create a mount target and these are the two mount targets that got created one for the uswest 2a zone and one for us os 2b the next step is to actually um provide the information about this newly created efs file system to kubernetes by configuring um a persistent volume and a persistent volume claim and kubernetes is able to do that through the use of a container storage interface driver for efs so this second script will deploy the csi driver to kubernetes you will see the csi controller and nodes running in the cluster then it creates a persistent volume a storage class called efs storage class a persistent volume called efspv and persistent volume claim called efs pvc which will enable us to mount this shared file system into any of the pods when all of the containers enter the running state we will see that this persistent volume claim will become bound to the volume and available to all of the pods so um if i uh get the current the available pv we now see that the pvc is bound and ready for us to use next we will just use our cloud9 ide to build a deep learning container and push that to the elastic container registry for that i will just need this one screen and we'll execute step number three there are just two simple scripts here three one will build the container and while it is running the docker build let me walk through what the actual build is doing it's using this docker file which takes the base image of ubuntu 2004 installs pytorch in this container image and then just copies all of the code that imran walked through earlier uh into the container image in as well as a bash script called data prep which we're going to use to download the data from the web and we are at uh step 4 out of 10 in the building of the container basically we're installing um by torch and various dependent packages and then the rest of the steps are pretty quick and our container is the container images is done building so in order to use this image from the cluster we need to share it with all of our nodes in the cluster and that is done by pushing the image to the elastic container registry in the account so this is what we're doing with the second script in this step the we can also see the elastic container registry by typing ecr in the console i clicked on eks so this shows us the cluster and ecr will show us the container registry and this is the container that we just pushed into acr so as a next step we are going to run a pod that will use this image we just pushed and run it on one of the nodes and execute the data prep script which does nothing but w get the sci-fi data set and store it in the shared efs volume to do that we will cd to step number four and i will launch the pod by executing the first script you'll see this uh efa vfs data prep point being created if we take a look at the status it is container creating uh but that is because now the image being pulled from our container registry down to one of the nodes in the cluster so that it can get executed so now that it is in status running we can see that the container was created and started successfully and it doesn't take a very long time the image is now the images data set has been downloaded and the container has completed its work if we take a look at the log files that it printed out we'll basically see that it successfully downloaded the data set and saved it in this file in the shared volume so now that we have the data we've set up the shared volume and we have the elastic kubernetes elastic job controller in kubernetes we are ready to run our distributed model training job which is what we'll do in this step we'll see that the job will create a worker pod in each of the nodes and these worker pods will work together to complete the training so for this part of the of the workshop i'm only going to focus on the default namespace where the distributed training will be running so that we can better see the output of our jobs so this step will be executed in folder five and as a first script we will generate the elastic job manifest an elastic job manifest is basically a custom resource in kubernetes that's of type elastic job here we're giving it a name of cypher 10 train and we're configuring it to use the lcd service that we've deployed locally in the cluster for coordination of the workers and we're defining the job so that we would start with two workers and we'll run those on c5 forex large instances and use the image that we pushed to our elastic container registry the command that we're running in each of those images is torch run and the model training python script that imran walked through earlier we're going to run the training for 10 epochs and we will save the model file in this location while accessing the shared data in the efs shared volume the shared volume is mounted using our persistent volume claim called efs pvc and it is mounted in this path as we have specified here so running the job actually applies this manifest and as you can see that creates two workers one per each node and since we run the download job which is the same container the image is already pulled onto one of the nodes so it becomes running immediately however the second node still needs to pull the image from the container registry so we'll just wait for that to happen and now it begins running at this point we can take a look at the logs that these two worker pods are putting out and i i'm using a tool called cube tail here which aggregates uh the logs from each of those pods and prints them on the same screen color coded so you can see now that both pods worker 0 and worker 1 are in fact doing the distributed training together and printing out the loss of the training on the screen so you will see two numbers here between the brackets the first number is the number of the epoch and the second number is the number of the iteration inside of the epoch if we were running this job on a single worker there would be 400 iterations however because we're splitting the work between two pods each of the pods needs to do 200 iterations in an epoch and that way we're doing the work twice as fast essentially the uh epoch goes up to 200 and then a checkpoint is saved and we continue with the next epoch a good a way to demonstrate the elasticity of this distributed training job is to demonstrate what happens if let's say we're using um spot instances in our cluster or if something goes wrong with the infrastructure and the nodes get terminated mistakenly then um the job will completely recover by restarting the pods and continuing the training from the latest safe checkpoint i'll demonstrate this by deleting the current worker pods and letting kubernetes recover the training job so the cube ctl delete pod and i'll just paste worker 0 and worker 1 here so what happens now is uh these pods get terminated but we still have the elastic job telling kubernetes that we want to have two workers running so kubernetes automatically creates new worker pods and if we take a look at the logs we see that there is a new rendezvous happening between the two pods they're coordinating the work between themselves using the lcd endpoint and they find the latest checkpoint that was saved and training continues uh from uh the latest saved epoch which uh happened to be number five so we're now continuing to work on epoch number six um one [Music] other detail that i wanted to mention here is that we can run the utilization script and see that actually the two nodes are actively participating in the training and we see cpu utilization and memory utilization in both of the nodes the training takes about six to seven minutes for 10 epochs and what we can see here is that the loss gradually decreases as the model becomes better and better through this training to get a really accurate model usually 10 epochs are not sufficient and we need to train the model for longer but for the purposes of this workshop we'll just train up to 10 epochs and while the training is finishing i want to mention a couple of details usually in production you would see distributed training happen on gpus instead of cpus as we're demonstrating here just for the purposes of the workshop also the distributed file system would normally not be fest but fsx for luster which is a file system that provides much higher scalability and throughput specifically for distributed training workloads one caveat for fsx is that it only supports a single availability zone which is important and usually the case when we're doing high performance training and want all of the nodes to be located as closely as possible together so now the training is finished and we are ready to execute the next step which is to see how good our model actually is for this we will create a kubernetes job which will run a pod which will provide the model that we just created some new data that it has not used for the training and we will see how good the model classifies this new data into different categories so let me scroll here to step six and we will execute the generate job script first to create this kubernetes job manifest called scifir10 test which will also run on the c5x c54x large instance and it will use the same image that we have been using so far and the command we're running inside the the container is just python and then executing the cipher 10 model test script passing the saved model and the location of the test data from the shared efs volume we're again mounting that efs volume using the persistent volume claim the same way as we did for the training so let's apply this manifest we will see this nuke container cipher 10 test start running here immediately and it completes pretty quickly because this is just an inference job for the model not very resource intensive and we can take a look at the logs that the test job printed out so we've provided 10 000 images for testing to the model and overall it was able to classify these images with accuracy of 75 and these are the various categories that it categorized the images in it looks like this model is worst at being able to recognize cats for some reason but it is actually best in being able to recognize trucks so that is um the test for our model i want to point out that we have a extra step for cleanup if you wanted to decommission all of the infrastructure that we created so far that is done in two steps first we would destroy the shared efs file system and secondly we'll run an eks ctl command to delete the cluster we are not going to execute this step here but that is how you would do it if you wanted to decommission the infrastructure so i would like to go over uh these key takeaways from the workshop first we used aws elastic kubernetes service to orchestrate and run a distributed training workload we have two more workshops coming up that essentially do distributed model training using different services in aws one would be doing that on amazon sage maker and the other would be doing that on adobe's parallel cluster regardless of which orchestration mechanism you choose the principles of distributed model training would be similar for distributed model training we use the pytorch ml framework and torch elastic for kubernetes and finally the architecture that we displayed in this workshop can be used for your own use cases and you can access the code for it for an end-to-end um description of all the steps and you can borrow the source code from github which is available at this short link bitly dtw hyphen eks i want to take just one moment to say thank you to all of our collaborators who helped put together this content um especially um the uh members of the frameworks team and our colleagues from mehta hamid missouri and matthias russell thank you for watching and if there are any further questions please feel free to reach out to us either on github or linkedin have a great day you