Distributed ML training with PyTorch and Amazon SageMaker - AWS Virtual Workshop

Key Takeaways

presenter for today Shashank. So, the first thing we'll do before we get started is to get our accounts set up. Uh We'll click on the link that's on your uh on your chat box there. That It should say eventbox.dev/code and a number. Click on the link. This is the very first thing you need to do. This will take you to a website that looks like the one that you see on my screen today. Um it's a website that has all the workshop material. We'll walk through it. I'll go through the process of uh um uh walking through the workshop material, also getting your temporary AWS account, and all that. But before you do that, just click on the link and get started. Um again, welcome folks from uh from India, from from Canada. It's amazing. Minneapolis. It's amazing. Uh Yeah, so I'm assuming you can see this website, so I'm going to start with the workshop material. Okay? Uh we are 11 minutes in, so I think we should get started. All right. So, uh welcome to this uh workshop PyTorch distributed training with Amazon SageMaker. Um I'm assuming you have some familiarity with PyTorch, which is the main framework of choice for this. Uh if you're not familiar with SageMaker, that's totally fine. We'll walk through what SageMaker is and how you can you train PyTorch models on SageMaker and also train it in a distributed fashion. We'll talk about what distributed training is, when you should use it, uh and there are a few different ways to go about doing this with PyTorch and SageMaker, so we'll discuss that. Uh as I mentioned, this is sort of a um sort of an advanced topic, if you will, uh but we'll try to introduce it in a way that that makes it uh easy for you to get started. Uh even if you don't have any familiarity with uh distribution training as such, we'll share all the tools that are available to you, so you can take those tools and apply them to your models and your work, okay? The primary focus here is to not to teach you the science or the theory that goes into why things work, but to give you a good understanding of the tools that are available to you, just like PyTorch, just like SageMaker, so that you can be productive with using these for your work, uh whether it's school or research or or your your day job, okay? So, we'll get started here. Um yeah, awesome. Thanks uh for confirming. Um I don't I won't uh bring out names here, but thanks for confirming that uh you can see the website, and uh welcome from Mexico. Welcome. Uh so happy to have folks from all over the world. Okay, so in this website, the first thing you'll notice is this uh link at the very bottom here. So, this click on this link, this link will guide you to the process of uh uh getting a temporary AWS account. Actually, before you click on it, um there's instructions on how to how to go through that process. So, if you click on getting started on the left, it should open up with two different sub options here. Section 1.1 on the left uh goes through step-by-step process with screenshots on how you can get your temporary AWS account. So, it's guiding you to the you to the process of clicking on the link, and you'll be greeted with a page that uh has a pre-populated code. Don't change anything there. Simply click accept and continue, and then it'll ask you to sign in, and the sign in uh could be your um uh one-time uh your email address, where it'll send you a password code. In my experience, it takes like a minute a minute and and half, so be patient. So, click on it, give your email address, wait for the code, and once you get the code, uh you fill it into the into the account, and then uh you once you do that, there'll be a button that says AWS console. Uh you click on that, and yeah, it'll open up the console, and uh boom, you have your AWS account, okay? The whole process should take no more than a few minutes. Uh the biggest uh lag here would be the duration of time it takes to get your code when you give your email address. So, get started with this section. So, section 1.1 uh walks you through the process of getting your temporary AWS account. This is key. So, I will walk through the rest of the AWS uh rest of the interface a little later on. Right now, just go through this process. Uh I have kept the uh I I've made sure that the accounts are already ready, and they have SageMaker uh notebook in uh studio notebooks already ready and running. So, when you log in, uh I'll walk you through in the section 1.2. When you log in, you should have everything ready there uh to get started. So, no no setup is required. Uh it's already pre-warmed up and set up. So, go through this process for me now. Uh again, section 1.1, click on getting started, and then click on section 1.1, and follow the screenshots to get access to your temporary AWS account. Now, if you see any issues in the same Q&A chat where you've been posting um uh you know, where you're where you're uh where you're joining in from, just post your uh questions there, post your comments there. And sometimes for me, uh even comments about something worked or something uh I was able to get to this stage or something like that is helpful, so I know that um at least some of you, um, if there's some issue, or if there's an issue that's affecting everyone, or if there's a specific issue to for few people. So, just saying, uh, "Hey, I was able to get to it. I was able to open the account." Stuff like that is helpful for me. So, I'm watching the questions live, so feel free to post them there. Uh, go through section 1.1, and, uh, let me know if you have any questions, and, um, uh, yeah, there's uh, there's no small question. Just post all questions you get, because others may be, um, seeing similar questions, too. So, have similar questions, and I can answer them live. Awesome. Come on in, more folks from, uh, from from, uh, California. We have more folks, uh, who've just started. So, yeah, if you just started, uh, click on the link that's in the Q&A that says eventbox.dev/something, such a code. Click on it. It should open up a website that looks like the one you see on my screen right now. And what we're doing here is going through section 1.1. That gets you to the process of getting your temporary AWS account for this workshop. Okay? Go through the process of section 1.1, follow the instructions, uh, click on the link, and go through this, uh, these steps, and you should get an AWS account. And, um, if anyone, uh, has is if one of you've been able to go through the entire, um, process, and have your account ready, just share in the Q&A, so I know that you have access to your account. And, uh, we'll wait a couple more minutes for you to go through this process now, because, uh, you need your account to do the workshop. And, uh, uh, we're making it easy for you with a temporary account rather than bring your own, so go through this process and we'll uh we'll uh get started with the material soon. And welcome everyone again. Thanks for joining. Um always nice to see folks uh joining in from around the world. Thanks for being here. Awesome. Okay, I got my first confirmation that uh they're on the notebook. So, um all this all the uh all the instructions are hopefully clear then. Uh okay, so I'll wait uh while whiles the rest of you go through the process of section 1.1, um I'm just going to briefly go through section 1.2 on the live stream just to talk about uh how to download the workshop material. Uh this whole section getting started is about getting your account and getting your material. So, in section 1.2, uh if you're familiar with Jupyter notebooks, then you should feel at home because this is all this is is a managed Jupyter notebook uh uh service on the cloud, right? So, you don't have to manage your own, you know, service and stuff like that. So, this notebook interface should be very familiar to you. So, go through the process of getting the material and to do that, you'll click on file, open up a new terminal window, and uh clone the material. So, there's a command there in the in the uh workshop website, and I'm going to do this live. So, I'm going to be doing most of the workshop live with you so that um we're following the same steps and uh if you're stuck somewhere, you can also see this. So, I'm going to copy it, and I'm going to clone the repository as I'm doing right now. Again, if you see issues with uh font size not too big, you can't see my screen, just post them on the Q&A so I have a sense of uh uh where you are right now. So, I'm going through the process right now. So, clone the repository. Uh it says double click, open the notebooks. So, double click. Open the notebooks. You should see two notebooks there. And then you click on the first notebook. And then you need to choose the PyTorch kernel that I specify here. Uh choose the CPU kernel. Don't choose the GPU kernel. Um So, PyTorch 1.8 Python 3.6, okay? And say select. And that'll take a a couple of minutes just to get the um the instance uh for this notebook ready. So, that'll start the notebook kernel. Um behind the scenes it's actually provisioning this uh instance for you that where you'll be running this code. Okay? So, if you're done with section 1.1, go through section 1.2. This whole thing should take 20 minutes net or less actually. Um maybe 10 minutes, 10 to 15 minutes. So, uh go through that and if you have questions, of course, um feel free to share on the Q&A, okay? We have more folks just joining in right now. All right. Um all right. Good, good, good. Um good. Okay. Let me know. We'll wait a couple more minutes as you go through the process of one securing your temporary AWS account and then downloading the workshop material uh uh, onto this notebook instance in SageMaker. Um, in section one, um, yeah, if you're stuck anywhere, let me know. Some of the, uh, console interface sometimes changes. Um, it just, uh, improves the improve the usability. Sometimes buttons move in different places. So, you can always, uh, click on, uh, the search bar on top and search for SageMaker or search for any service to get access to, um, service and, uh, if some of the, uh, buttons have moved around, then it's possible, but, uh, let me know so I can go through the, uh, experience with you on screen so I can guide you if if you're stuck anywhere. Okay. There is an issue with, uh, someone saying, um, "I'm I'm unable to open, um, I'm unable to open studio." So, you should be able to, uh, uh, hold on. Um, give me a second. So, you should be able to, uh, try now get the, give me a second. So, click on the the some of the user interface may look a little different. Sometimes they do move things around. So, if you click on, uh, SageMaker Studio and then click on on launch studio, you should see you should see a user and you should see an, uh, launch studio or open studio button there. Uh, Is is anyone able to get to the Jupiter notebook stage? I think I saw some, yeah, I I I think I see some folks are on the Jupiter notebook stage. So, don't worry about settings, other settings on the on the Jupiter on the notebooks. Just make sure you're able to come to the notebook instances a notebook here, studio notebook, and that's all you need access to today. So, don't worry about projects and other things for today's workshop. Just make sure you're able to get to the studio notebooks and we'll go through the rest of the workshop here in a moment. Couple more minutes as uh folks go through this. So, I'll just um maybe just sort of highlight the process. So, you should see something like this when you go into the SageMaker console. And you can click on studio. Few of the links have changed. They change the user experience sometimes. So, click on studio. Um launch SageMaker studio and you should see you should see a user already pre-populated like this. Okay? And when you click launch app, you can select studio. So, just select studio. So, some of the user interface has evolved, but click on studio. So, so, if you um have access to the console, you should go through this process and be able to launch your studio notebooks. Okay? And again, thanks for your question. Just uh let us know if you run into uh Okay, good. Good. So, folks are able to run through the notebook. Um yeah. Yeah, someone else mentioned uh the interface is different. Yes, I think they pushed uh recently pushed like a console um update where the user interface changed and sometimes they ask you if you want the new console experience or the old console experience. So, depending on which you chose, um sometimes you'll get a new console experience and uh some of the uh some of the uh buttons may look a little different, but in essence, go to the studio, launch studio app, and click on uh studio here. So, you should I preconfigured all your accounts with a user, so you all should have a user and the studio should already be running. So, if you were to do this from scratch, it would take uh quite a bit of time to actually launch the studio domain and get access to a notebook. So, these are preconfigured, so just launch your just launch your uh notebooks, and uh I'm happy to see some some of you are inside. So, you're already through section um 1.1. Uh go to section 1.2 to get access to your notebooks, just like I did. I went through the process of uh uh cloning this uh Git repository. The link to it is already in the in the workshop website. And once you clone it, go to the notebooks folder, and you should see notebooks. Okay? And uh if you're wondering what else is in this uh GitHub repository, basically, all the all the stuff required to host this website is in that GitHub repository. Uh so, if you were to use this uh workshop website after the workshop, then you could, you know, run it on your own laptop or desktop. But, uh for now, just focus on the notebooks folder, and we'll get started. Okay? Couple more minutes, and we're in. So, through section 1.2. If anyone's done with the entire section one, 1.1 and 1.2, let me know in the in the Q&A. Uh let me know in the Q&A and we will we will uh proceed to the next section. So, a helpful tip from one of our attendees here, um they said they uh preferred using the search bar on the console to get to studio rather than the screenshot, so you can try that. So, you can always use the search bar on top uh of the SageMaker console and you can always use that to uh go to studio or any service directly. So, yeah, use the search bar. Search bar is awesome. Search is awesome. Okay? So, if some of the screenshots uh look different, uh use the search bar and you'll get to where you need to be. Awesome. All right. We have Awesome. So, I'm getting some confirmation in the chat that uh um you're done with section 1. Okay, let's get started with section one section two, which is the meat of this uh which is the meat of this uh uh workshop uh and I'll mention a few things about the workshop interface, of course. So, we jumped right into getting your account because that's important. It takes a little bit of time, as you can see, it took about 20 minutes for most of you to get here. So, the workshop interface itself um I'll just walk through some of the few elements on the screen where you can provide feedback and get support and stuff like that. Uh you have your uh chat here your Q&A here where you can share questions with me and I'm happy to see them and answer them for you all live stream. On the workshop platform itself, the way you navigate is there are sections on the left and there are subsections. Just click on the links and you can navigate through these. And the blue button you see at the bottom of the screen that says workshop support feedback, so you can also use this to uh provide you know, thumbs up, thumbs down feedback. If you have It's helpful if you give sort of explanation on what's working, what's not working. On the Q&A out here so that um Pardon me. So, if you share feedback here, I won't be able to see it right now, but if you share feedback on the Q&A, I can see it right now and answer your questions, okay? And uh Excuse me. So, that's sort of the brief high-level overview. So, we'll move on to section two and we'll go through the workshop material today and the code example what we have and how you can run distributed training. Okay, so here's what we'll cover today. Okay, while you go through the while some of you go through the setup, we'll just go through the steps here. So, we'll talk about um what is distributed training, why distributed training, we'll talk about what is Amazon SageMaker, and how do you use Amazon SageMaker with your favorite with your favorite framework. In this case, it is PyTorch, which is what you're you've come here to learn about here using the SageMaker. And then, we'll talk about how you can use SageMaker SDK and SageMaker to be able to run training using your PyTorch code, okay? So, we assume you're bringing in your PyTorch script and how you can train this at scale with Amazon SageMaker and we'll go through some notebook examples that you have access. So, let's start with some high-level why distributed training, right? So, why would you consider distributed training? In most cases, what is what is your primary goal for training? Your primary objective is to reduce time to train, right? You want to get results faster, and you want to train uh faster. So, if if you're taking a week and if you can do it in a day, then it's preferable, right? So, that's your primary objective. How much time it takes to train a model to reach a desired accuracy, right? And there are different ways to go about speeding up training, okay? You can obviously optimize your code, but there are um at some point you there's there's only so much you can do with your code, and you want to provide more compute, right? Can I give you more compute power so you can train faster? So, there are two different paradigms here. So, scaling up is a matter of uh if you're training on a specific instance and uh you replace it with a more uh powerful uh CPU or a GPU, then that's called scaling up. So, all you're doing is merely replacing what you have with something bigger, so it trains faster, okay? This is always a preferable method before you go to distributed training, okay? If you are training on a smaller GPU and you can get speedups with bigger GPU, you should do that. Uh scaling up is always preferred, and once you hit a wall with scaling up, as in you are using the most powerful in um CPU or GPU you can, and you still need more performance, then you can go to scaling out. Scaling out is a process of taking your uh data set and code and using multiple machines to train faster, okay? So, these are sort of the two paradigms for getting faster results. So, your goal is very important. Your goal is to get faster results, and there are two different approaches. So, always scale up before you scale out. Of course, this workshop is about distributed training, so we are going to be talking about scaling out, which is how do you use multiple CPUs or GPUs in order to get performance benefits, right? So, with that in mind, um a quick overview of sort of um how distributed training works and why it speeds up, okay? In your typical training, you have your uh uh your data set, you break them up into batches, and I'm talking about deep learning specifically here. And these batches go through uh the training process, right? You go through the forward pass, backward pass of neural network, which is running on a single instance. Now, the bottleneck here is of course that you these batches have to go through sequentially, right? They go through sequentially. Now, when you have more compute or more CPUs or more GPUs, you can actually spread these batches to different uh instances, uh in which way you can have copies of your model on all these different uh instances. By instances, I mean like this, CPU instances or a GPU instance on AWS. And because they see different batches, effectively you're training on a larger batch size at the same time. So, in the first step, you have to go through this in sequential order, and in the distributed setting, you'll be able to process a lot more batches at the same time. Okay? So, it it sounds simple in theory, but there are few changes that you need to make uh to your code often um just because of the complexity of being able to manage multiple CPUs and multiple GPUs in the cloud, okay? So, let me before I move on, uh let's take a look at uh if there are any questions available. Pardon me. Um Need some water here. Okay. Yeah, some questions on uh how how long the accounts will be alive. So, I've given a cushion out of 2 hours. So, at the end of the workshop, 2 hours after the workshop, you'll be able to continue using the account. You know, you want to play around with the code and options and such. Okay? Okay. So, yeah, before we jump in. So, there are a few different approaches to distributing your training. So, back in the day, like one of the first approaches was parameter server approach. And if you remember the days of Google had published a paper, DistBelief, that is the internal system that uses approach. Basically, all your workers could be either CPUs or GPUs, and you have a server, a central server that manages all the averaging of gradients and stuff like that. So, you had this central node that was sort of a bottleneck, if you will. So, which was responsible for um taking care of all the different workers, and they would like run through the data set and give gradients, and the gradients were averaged in the in the parameter server, right? Since then, newer approaches have come into play, such as ring all reduce. Basically, you remove this central authority, if you will. And all the GPUs or all the CPUs can communicate with each other in order to perform an all reduce operation. An all reduce operation is a way to average your gradients across all the different nodes or all the different processes. And there are a few different ways to go about So, this is in theory, right? But, there are a few optimizations you can do in order to get the same results faster. And PyTorch has several different backends where you can achieve some of these things in a faster way. So, there are a few optimizations it performs that I won't go into too much detail, where you don't have to often um wait for the entire uh gradient computation in order to uh average the gradients. So, there are a few optimizations where it can do some of the some of the averaging in in parallel to some of the uh forward pass and backward pass computation that's happening on the instance. So, there's the there's some uh optimizations that are already available in the PyTorch library just by choosing some of the backends that we'll talk about. And that'll help you get further speed up compared to um sort of naive implementations, if you will. Okay? So, you don't need to worry about how it is doing it in this workshop, too. We'll just focus on the tools that are available to you. The science is interesting. There's a paper out there from uh Facebook that goes over how this is done. Uh there are other papers that talk about ring all reduce, like the Horovod paper that came out a few years ago. If you're interested, there are um the The science is interesting, but for us, most of us, I think what we want to focus on is the implementation and how we can get um get productive with this, okay? So, uh again, before going to the code, I'll just talk about how PyTorch and SageMaker interact with each other, okay? So, you're familiar with PyTorch, it's a deep learning framework. You write code on it, you train models. And you may or may not be familiar with SageMaker. In this case, SageMaker is responsible for managing the infrastructure for you, okay? So, you bring in, if you look at this uh diagram in the in the web page in section 2.2, uh you you bring in your PyTorch training script, okay? You bring in your PyTorch training script, which is your uh you know, whatever code you have training with different model. And then, you bring in your uh SageMaker SDK API. So, there are a few different functions called estimator and fit. So, you write these two artifacts, okay? You're already familiar with writing PyTorch. You learn how to write these estimator functions in SageMaker. And once you do that, SageMaker is responsible for managing all the infrastructure for you. So, what happens behind the scene is you write this PyTorch script, and you write an estimator where you say, "I want you to train this on like eight GPUs or eight CPUs using specific versions of PyTorch, right?" And this is the location of my dataset. That's it. So, you provide this information in the estimator function. Behind the scenes, again, you don't have to worry about what's happening to the to the left of this red line. So, to the right of this white line, but here's what's happening. So, behind the scenes, it'll take your PyTorch training script. If you specified Python PyTorch version 1.8, then it'll go get the 1.8 version of the container, drop the script in, and then it'll go through the training on a on specified number of instances that you specified. So, if if you said you wanted this on eight GPUs, then it'll go run it on eight GPUs. Okay? And it'll also manage your dataset movement from Amazon S3 to the training instances and back again. Also, save your models and other metadata back. Okay? So, this is sort of like I describe it as a shared responsibility model. So, you are responsible for writing PyTorch script train script. You are responsible for updating your training script to do distributed training, and we'll talk about what those updates are in the in the code example. And then you are responsible for writing the SageMaker SDK estimator and fit functions, okay? You are responsible for that. SageMaker is responsible for taking care of the infrastructure. It's responsible for the data movement that happens between S3 and those training instances and also backing up your checkpoints. And this is at a high level, SageMaker has so much more capabilities. It has ability to debug models. There's you know, orchestrate pipelines with sort of like your CICD. It's It's It's a large service with many many many different capabilities. But at a at a very high level, it takes care of the infrastructure for you in a managed way. And you as a as a developer data scientist focus on your focus on your uh uh you know, PyTorch training and training script. And this will become evident when we run the example, okay? So, section 2.3 is where we'll jump into the workshop code, okay? So, open up the first notebook. Some of you have already opened it up and you're inside. So, open up the first notebook and now let's let me go through it and you can follow along with me. Okay, you can execute this section along with me. So, this is the first notebook and make sure you select the the specific version of Py uh PyTorch I mentioned here. So, this is section 1. PyTorch 1.8 Python 3.6 CPU optimized, okay? Now, SageMaker Studio gives you access to many different backends. We won't be changing with it, but just to give you um an overview of what are available. So, you can actually switch backends for your notebooks. So, this notebook is of course your your traditional Jupyter notebook, but unlike your traditional Jupyter notebook that you run on your laptop where your hardware is fixed, right? You have your Jupyter notebook client and your hardware is fixed. Here you can switch the hardware. So, right now it's on a CPU instance. I can pick a larger CPU instance or I can pick a GPU instance, okay? I can switch I can switch my backend. I can switch my hardware here. So, that's pretty cool. So, let's say you're training locally, if you're running experiment a script on a CPU and you want to try it on GPU, just switch the back end and you can do it. Um on your temporary accounts, you won't have access to a lot of GPU instances. So, let's not try that now. Uh you can give it a shot after the workshop because I'll keep the account uh available and running. For now, let's go through this uh example. So, I'm calling it PyTorch uh native distributed training with Amazon SageMaker because PyTorch natively offers few different um back ends and uh uh I'll show it to you in code so it becomes clearer. Okay? So, first thing we'll do is some uh housekeeping, right? So, run this section of code here and if you're unfamiliar with Jupyter notebooks, you can run click on a section and click on this play button. It'll execute uh these cells. Uh basically goes through the Python code and executes them and if there is any output, it would display it at the bottom of the cell. Okay? So, there are few housekeeping stuff here and I'll just briefly go over them. So, you need access to few SageMaker uh sessions and access to the client. So, we're going to do that here and you also need access to an execution role. This is required if you're new to Amazon, this is required for SageMaker to be able to access other services like S3. So, everything is permissions based now. SageMaker needs to have permissions to access data sets in S3 to do specific actions like read or write and stuff like that. So, the role gives SageMaker those abilities and in this case, I've already set it up. So, don't you don't have to worry about it. And um I'll also share uh get a default bucket to save my S3 models in SageMaker. So, there's there's a bucket and I'm going to just uh use the default one. You can specify. Now, for the specific code example, we'll be using the CIFAR-10 data set. It's a pretty simple model, uh a step up from MNIST, but nothing like the models you're probably training in your day day-to-day job or your research. So, this is a um smaller data set with 60,000 images, small, you know, 32 by 32 uh pixel color images. So, I'm going to use Torch's uh native data sets uh capability to download this data set, and I'm going to download this locally onto this notebook instance, and then upload it to S3, okay? In your case, you may have your data set already in S3, or you may upload it to S3 via different mechanism, but for purpose of uh this demo, we'll download this data set. It'll take a couple of seconds, and once it downloads, uh we'll push it to upload we'll use this upload data function to push it to Amazon S3, okay? And as I'm going through this, feel free to go through this, and if you run into anything, let me know uh in the in the questions. You've been great at asking these questions so far today. So, uh please put them in the Q&A, so I know that you're making progress, and I know that you're not stuck anywhere, and um I know that I can help you and others who have similar issues should run into issues. Okay? So, as soon as it uh did that, it's uh it's uploaded the So, here's the location it uploaded the data set to, okay? And you can go to your SageMaker account, and you can actually um you can go to your Amazon S3 here, and you can take a look at the the account, okay? So, uh take a look at the data set that is available on S3. All right. So, I won't do that now, so let's go through the next section, okay? So, the next section is where you'll specify hyper parameters. Now, it's important to specify these hyper parameters um at this level and not hardcode them into your training script. And here's why. If you hardcode this into your training script, you can't run experiments using SageMaker. Let's say you want to change some of these variables and run a second distributed training job, then you won't be able to do that if you hardcode it into the training script. You'll have to go and modify that. Whereas here, I can merely change these numbers and then rerun this notebook and then I can spin up multiple such jobs. Um makes it easy to experiment. I can even run these training jobs in a loop that can span multiple values for these if you want to run sort of an experiment that goes through a uh what is the effect of the change in batch size on your training? That sort of thing, okay? So, I specify these hyper parameters up front here. And the third step here, uh I'm going to execute this cell. And the third and the next step here is to specify your PyTorch estimator function, okay? Um after this, we'll go into the PyTorch code now, okay? At this point, uh what you want to do is provide your PyTorch training script. And this is, you know, your PyTorch training script with few modifications. And we'll talk about what those modifications are. Okay, this is your PyTorch training script. And then you'll specify where where it is at. It's in the code directory. So, on the left you should see a directory called code. And this code directory should have the specified file that you see here. So, which is the CIFAR-10 distributed native CPU. .py, okay? This exact um this guy here. So, this guy here should be here, okay? Uh, in the code directory. So, you specify that code directory. And, uh, couple more options that are optional, like where you want to share the training job information and all that. And then, you go down this path and then, uh, you, uh, specify two important things here, okay? These two are very important. The first is instance count. So, since we're doing distributed training, we want to be able to train on large number of CPUs or GPUs, right? So, in the estimator function, you can specify how many you want. So, you can say I want two or four or eight or 16 or 32 or 64 or whatever it is, right? It's as easy as that. Just set this variable and SageMaker will spin up those instances, uh, those number of instances so that you can do training on them. And the instance type is what type of instance. If you're new to AWS, um, these are basically different, uh, types of instance. A C uh, C5.2X large is a CPU instance, a compute-optimized CPU instance type, okay? And, uh, there are GPU instance types that you can put strings here. And these strings are available, excuse me, are available on the, uh, on the SageMaker website where there's a whole list of, uh, which to choose, okay? And later, I'll share some resources on when to choose, how to choose the right GPUs and all that. So, for now, uh, instance count and instance type are the most important things here. How many you want in your distributed cluster and how many uh, what type of instance you want copies of. And then, you can specify your PyTorch framework version, your Python version, and pass in your hyper parameters, okay? And, uh, the last step, the last step here is is fitting the function, okay? Now, estimator.fit will fit your training will will kick start the training job, okay? This will instruct SageMaker to go spin up those instances and and So, this is covered in this sort of diagram here, right? It'll kick start the training instances and then it'll copy the data set from S3 and it'll, you know, save the model back in S3 and then so on and so forth. Okay, all that sets into motion when you call the fit function. So, I'm going to run this. So, the other information you provide here is train and data sets and data sets is the location of the data set in Amazon S3 that was the output of this upload data, right? When you say upload data, you get data set, so that is the location of your data set in Amazon S3. And I'm going to run this and now it'll kick start. So, it'll take a while, right? It's a distributed training job. Now, what we'll do is actually pause for a second and see if you have any questions and then we'll What we'll do next is go through the code uh and the changes that you need to make to go from a vanilla torch uh example into a distributed uh aware torch example and a SageMaker aware uh example, okay? There are a couple of changes, as I mentioned in the uh in the website here, how they work together. So, this is sort of a shared responsibility model wherein you write the PyTorch script, you update the PyTorch script, and you write the estimator functions, and SageMaker does the rest. And we'll go through what those changes are. So, I'm going to pause here. Uh we have another 40 minutes left. So, So, as we go through this, I'm going to pause here and ask if you have any questions. So, so far, give me a thumbs up or give me a say yay or yes if you've been able to walk through this notebook with me and be able to run it. Okay? Uh Give me an indication in the in the Q&A so we can uh so I I have a sense of where everyone is and how you're doing, okay? So this has kickstarted the training job right now. Um I don't see any activity right now here. So I'm assuming I'm assuming everything is going good. You all have access to uh you all have access to your uh notebook instances and you're able to run through the training script. Uh sorry, the notebooks and initiate the training. Okay? Um Let's give it a couple more seconds. If you have any questions, I'll just uh pause here for a moment. Uh give a breather for everyone and uh it's a short workshop, but you know, if you want to go get grab a cup of coffee or something like that, that's good. Uh Okay, I see some question Ah, awesome. Thank you. Thank you so much for your feedback. Uh you know, it's it's hard to do a virtual workshop when you don't get feedback, but thank you. I think a lot of you are saying yes, so you're able to go through this these steps and you're able to run the notebook. Um Yeah, I can run the notebook and uh some good feedback. Okay, good good. Um I don't think anyone's joined uh later and has missed some of the instructions, but uh yeah, the workshop website instructions are on the um on the Q&A that uh that we posted there and uh go through the getting started section. Everything should be self-contained there. Go through the steps to get your account and that. So, on my screen, you should see that uh the training has initiated and this is basically the output of the of SageMaker's output all the um configuration details are here. So, it's all um it's going to spew out it's going to spew out all the um you know the uh standard out that comes on those So, it's this code is running in two different containers on two different CPU instances right now, okay? And you didn't have to manage any of it of that. It's it it happened automatically for you, right? So, behind the scenes. Right? This is uh this is um happening behind the scenes for you. And then uh uh as it trains, you'll see two different colors here. Uh each color represents the training that's happening on each of the instances, which is why you see a blue and you see a pink. Now, if I were to say I want four instances or eight instances instead, you would see multiple colors there. So, it's training right now, right? You'll see here that there's epoch one um loss and you'll see each color represents the epoch one on specific um training instances. And it's going through this and it'll go through this I set uh I set the uh uh I set the number of epochs to 20, I believe, here up front. So, it'll go through 20 uh epochs, okay? Go through go through here 20 epochs. Okay, now let's go through the code example and see what changes you need to make, okay? As we go through that. Now, this is your vanilla again vanilla PyTorch script with few different additions. So, you do have a section here for um I you you you do import this uh torch distributed as this. So, this would be new to you. Um, this is used to prepare your uh model for prepare your model for you know distributed training. Some of the other things here should be straightforward. So, this is new for you and and this is new. So, we'll use this later to sort of wrap the model to to tell PyTorch that this is going to be sort of a distributed training approach. Now, after after we after importing that, you'll define your model and here's a very simple vanilla um you know, very simple neural network model. You could of course use something like ResNet 18 which which actually gives you much better results um but it's a much more complicated model than this. You can bring in your model, right? You can define your own models here. Okay? So, this is sort of the uh um your training your model definition here. And then I define few helper functions and these functions are just to load your data, do data augmentation um for training, right? So, there's uh some data augmentation stuff here uh in the transforms and then we define our data loader and this is used to load your CIFAR-10 dataset. Now, because CIFAR-10 is supported by torch vision vision package, we use that. If not, you'll write your own data loader in order to load depending on what format it is in, right? You'd you'd use data loader to load. Now, the interesting thing here is you see that uh so, your dataset was uploaded to Amazon S3, but this script is running in a container on an instance in the SageMaker provisioned instance, right? A cluster. So, how does it get access to this dataset? So, SageMaker will pass this training directory uh, information through to the script as a as a command line argument after it copies the data set from S3 to that instance. So, it before the training kicks off, it copies the data set from Amazon S3 into the instance where the container is running and mount that into the container and give you a local path to the data set. That local path is then fed into the script. So, SageMaker does all of this. You don't have to worry about it. So, it's doing the data set copy, mount that into the container, get a local path to the data set, and then provide it to the training script. So, in your training script, all you need to do is, um, specify this default location and the data set will be available there. And in our case, if I scroll all the way to the bottom, there is, uh, there is this this, uh, data dir command argument, right? So, SageMaker will pass this argument and in your code, all you need to do is use this data dir variable or or also this environment variable, and you can be sure that the data set is available there because SageMaker makes sure that the data set is copied to that container location, um, so that your training scripts have access to it. Now, um, I I am I've not done it here, but sometimes you have large models that you can't copy. SageMaker can also stream that data set. So, there's something called, uh, pipe mode wherein it can stream that data set. Uh, let's say you have 2 TB data set, uh, in S3. You can't copy all of it, obviously, to every single instance, uh, in which case you can stream the data set. It's called, uh, pipe mode, which is not covered here, but it's it's, uh, something you can use. Okay? And then, uh, we'll So, there's a distributed sampler, uh, because it runs in distributed training, um, for the sampler, we'll use a distributed sampler, uh, uh, using the train data set from the loader, okay? And here's here's where we create the loader, and then we pass the sampler here. So, we pass the train set, batch size, um and the sampler, okay? We just We're just defining our data set loader. We also define the test test loader uh because you want to evaluate the models. That's how you see the accuracy that I was showing you that. And then uh so, there's a step here because we're doing a multi-node CPU, there's a step for um averaging the gradients. And uh you won't know need to do this step if you're running this on on um on GPUs. So, the average gradient step will go through the process of uh taking all the So, you'll do a um forward pass, uh a backward pass, compute the gradients, and then average the gradients, and then use the gradients to update your weights, okay? Uh so, when you average the gradients, if your train is independently, you'll generate two different gradients, and you would update two different models. Whereas, once you get average the gradient, you use the common gradient to update the models, okay? So, that's the averaging part. And then finally, you have your own training loop here. So, in this training loop, uh again, couple of changes here. So, one of the changes here is to this whole section that says uh is uh is distributed. So, if it is a distributed training job, then we get access to sort of the world size is how many instances you have. Um if you had eight instances that you're distributing onto, then your world size is eight. So, there are eight instances in this world, okay? And rank is what Who am I uniquely? So, this script is going to run on all the eight instances, but how do I know what my number is, right? So, that is rank. Now, in I may be the second instance or the fourth Of course, it's arbitrary ordering. There is no ordering there, but it's it's Each script knows who they are uniquely, okay? This is important because there are some operations you only want to perform on one and not all of them. An example of that is if you want to save your checkpoint or save your model, you don't want to save eight copies of it. In this case, two copies of it. You just want to save it on one of them. So, you can use this rank information to know who am I uniquely and use that information to do specific things that you don't want everyone to do, okay? So, that's world size is how many instances you have and rank is who am I uniquely. Okay? And this uh This is where you initialize uh this init process group is is part of the distributed training package as well that's required to specify what your training back end is and your rank and world size. So, the this init process group Remember, it runs on all the instances, right? So, this helps uh these talk to each other. So, they are they know that they're part of this distributed training group and they need to communicate with each other and what kind of back end to use. So, PyTorch supports few different back ends. The back end that we are using here to do the distributed training. The back end that we are using here is called glue, which is commonly used for CPUs. Can also be used for GPU. Uh but if you want good performance with GPUs, NVIDIA offers a back end called nickel and PyTorch supports it, NCCL. So, you can use nickel. In this case, I'm doing a CPU only training, so I'm using glue, g l o o. And I believe I made this hyperparameter. So, you can make anything a hyper parameter, right? So you can actually switch So if I were to run this script on a GPU instance, I would switch the back end from glue to nickel. And then I would get better performance on GPUs. Moving back to our script here. So the that is the back end here and it's a command line argument argument that gets passed because a hyper parameter that you can consume it in your model. And this is the So we use back end glue or nickel. And there's a third back end called SageMaker data parallel distributed data parallel back end. Now this this back end is unique to SageMaker. We'll discuss this in the next demo, not in this one, in the next notebook. But I'll just mention briefly because SageMaker understands AWS infrastructure very well, there are some benefits to using the SageMaker data parallel back end when you're using large scale distributed training on SageMaker. Okay? They've We'll talk about how this works when we go to the next section. But for now don't worry about using the SageMaker back end because it only works on GPUs and only works on at least eight GPU instances. Okay? So it's not for everyone. It's for folks who are training those large models or on large data sets can use the SageMaker back end. But for a lot of smaller use cases, you can use glue or nickel that's available or natively available in PyTorch, which is why I call this distributed native as a as a file name. Okay? They're natively available in in PyTorch. Okay. No questions. I Okay, actually there's a a a few questions here. Let me finish this uh and we'll try to answer them, okay? Let me finish this. We'll try to answer them. Um Okay, after that we have uh I I after that I think it's just your vanilla PyTorch uh training. So, you get your train loader, your test loader, and then you get uh your model, and then you do this extra step of converting your model into a distributed data parallel model, okay? So, this is again instructing PyTorch that this model is being used in a distributed training fashion. The interesting thing to note is that the the Okay, I won't mention because it might uh confuse the train of thought. So, there there are other uh approaches to doing distributed training with PyTorch with that um NN data parallel, but I won't confuse you if you're not already familiar with it, but um let's let's just go through this script here. So, distributed data parallel, so this this will make sure that PyTorch is aware that this is a this is a distributed training job, and the training loops loop should look very familiar to you. No changes here. Uh there are the one change here is that uh if it's a distributed training job, then you do the gradient averaging. This is a function you you described um ahead there. And uh finally, there is uh uh another small change that you need to do which I mentioned earlier. So, saving models. Now, if you're doing distributed training on eight instances, as I mentioned, you don't want to save checkpoints or models on every one of them. You just want to save on one of them, right? Which one is arbitrary because there's no ordering to these number of instances. It's all arbitrary. So, pick the one uh pick uh so, I it's conventional conventional to pick zero, rank zero, uh which is like like {quote} {unquote} like a master or something, but there's really no master if you're if you're doing all reduce here. So, pick zero and say if you're in rank zero, by rank remember, rank is who am I uniquely, right? If I'm unique instance number zero, then I will save a model or save a checkpoint. Uh else don't save it, right? So, Um yeah, otherwise don't save it uh so that only one of the eight or in this case one of the two instances will save the model. All right? And then I have a test function defined, uh a save model function defined, um standard stuff. Uh I'll spend a couple more minutes here just to wrap up the code. So, I'm splitting the So, the script needs to be um needs to be capable of accepting some command line arguments to run with SageMaker, okay? So, I'm I split this into Python environments and then SageMaker environment, okay? The Python environment PyTorch environment are your hyper parameters, okay? Mostly. Um these are all the ones you specified here. So, epochs, learning rate, momentum, batch size, back end type, and so on and so forth, right? So, these are your These are your uh command line arguments that can be passed that SageMaker will pass to this script when it launches. So, remember what happens. It'll spin up a container on if you say eight eight instances, and when it spins up the container, it'll run your script. It'll run the script and it'll pass these It'll take your hyper parameters from the estimator function and pass it to the script, okay? So, these are all your PyTorch environments, which we consume in the code like batch size and all that. And then there are some SageMaker specific environments, and these are required for passing your data set like uh model data dir. So, these are automatically passed. You just need to define them here, and SageMaker will automatically know where your data set is in S3 and pass it to uh pass it to the training script. It'll also pass it it passes the it copies the data set from S3 to the containers instance, mount it onto container, get a local path, and then passes it into the script. So, it does all this behind the scenes for you. And also the model directory is like where you want to save the model in Amazon S3. Okay, eventually it wants you wants save the model as a backup, right? And hosts and current hosts are uh only for distributed training so that you know how many hosts are there and who the current host is. So, this is basically a world size and your rank, which is like who am I. So, SageMaker will again uniquely assign a number to each of your instances. Um if you've done distributed training in other um other setting like, you know, Kubernetes or something like that, you you can see how much of it is abstracted away. You don't have to do manage all of that, right? So, a lot of it is abstracted away. Right, it's all much easier and simpler for you to do um as you go through this, right? So, let's see the progress of our training. You should see the training should have completed by now, and you should see um result here. Uh hopefully you're all able to see this as well. So, uh it finished training and uh I I see average accuracy is about 60%, not great because uh very simple model. You can upgrade to ResNet 18, probably get around 80%, and there are some custom models that'll get you 90 596%. Uh you can find these architectures on online. So, um yeah, but for illustration, so you get about 60% accuracy here, and then it says um you know, up uploading generated model uh training model. So, because I saved it locally in the save model, SageMaker will take the model and upload it to Amazon S3, okay? It does this automatically for you. And then finally the This is interesting, the training seconds and billable seconds. So, the interesting thing about SageMaker is you'll only pay for the instances where you did your training on for the duration of training. Uh and you're not managing it. So, it's unlike say if you were training on EC2, you're paying for the EC2 instance every second you're using it. Whereas with the SageMaker, it spins up an instance when you call the fit function, your clock starts, and when the training is done, it'll tear it down, and your clock stops, and you're only charged for the duration it was running. So, this is like a nice um benefit. So, you don't spend a lot on computer resources when you're not using them. Uh especially if you were Say you got access to an EC2 instance for training, and then you just keep it running, and then you're just check paying for it even though you're not training, okay? So, uh that's good. So, that's I'll pause here again, and I think I'll try to answer some of the questions here. So, there's uh The fit function is throwing some error. Looks like it's looking for a GPU instance. It's unable to So, don't run the second notebook. I think you're running the second notebook. So, the second notebook uh will not work on these temporary accounts. I provided it as an example, and I'll go through it. It will not work on your instances because your instances only have CPU instances and have single GPU instances. Um it does not have access to the instances that are needed for the second notebook. So, don't run it. The first notebook should not ask you for GPUs because we are only using CPUs. Uh if you if you if you remember up top, we only specify C5 um where is that? I We only specify C C5 2X large, which is a CPU instance. So, you should not see you should not see a GPU uh error. Uh Oh, I think you corrected yourself. Okay, no worries. No worries at all. Then uh Okay, so another question is what was the feature that to stream data set when big data cannot be loaded locally? It's called SageMaker pipe mode. p i p e pipe, like a pipe mode m o d Um the default is called file mode, that's called pipe mode. You need to update your data loader in order to be able to do that. Uh there's a couple more lines of code change you need to do that. Okay, so let's move on. So, um the whole point of training is to host a model, right? So, you want to be able to host your model so that you can consume it. So, the next part of this uh will take the trained model that you just trained and host it. Now, if you see here, the estimator.model_data is the Uh let's take a look at what that that is actually. So, let's go run this estimator. So, we just finished training and estimator.model_data will tell you the location of will tell you the location of your uh uh uh It will tell you the location of your model, right? So, S3 SageMaker US East 1 my account number jobs. These are all temporary accounts. Um so, SageMaker native model.tar.gz. So, SageMaker automatically tar your model and put it to S3 and this is the location. And now what we'll do is we'll create a PyTorch model. So, SageMaker has um ability to host model endpoints that you can invoke. So, you can have like mobile devices or, you know, websites or other applications uh send requests and get back inference results. So, we'll do that here. So, I'm going to create a PyTorch model with that model data and I have an you need an entry point file called inference.py. So, this this inference.py basically defines the model definition. So, when SageMaker saves your model let's take a look at the save model function here. So, in our training script we save this model here. Uh all the way Okay, here. So, when it says save model, uh we're saving the state dictionary, right? It You were saving the state dictionary, but we don't save the model architecture definition itself, right? Which is why you need to provide both. So, the model has the state dictionary, you also need to provide the architecture definition so PyTorch can reconcile those two things. So, we provide this inference.py where we describe the model definition and then we load the state dictionary, right? From the saved model. So, that's you need that that script and we provide that script and then when you say model.deploy, SageMaker will now host an endpoint, okay? It'll now host an endpoint as soon as this this loading thing here is done and then we can ping the endpoint, submit some requests and get back results, okay? We'll do that. So, I'll pause here. If you're running this, go through this this section, run this code. Uh if you have any questions again if you have any questions again, yeah, feel free to Yeah, feel free to let us know if you have any questions and we will uh pause here just a moment just to make sure that there any questions try to answer them for everyone here. No questions so far. And also, yeah, let us know if let me know if everything's going good and uh uh everything's going good and you're able to Excuse me. And you're able to go through all these steps. Okay. Some silence here. Okay. Thanks uh Thanks, I think um Matthias replied with the pipe mode for the question about how to stream data set. Yeah, that's the one. That's the one. You want to be able to Okay. All right. So, my model is hosted now. So, you see this exclamation here. Uh you So, SageMaker notebook Studio notebook also gives you tools to observe all of this right within uh Studio without having to leave the Studio interface. So, on the left um I breeze through this quickly. So, if you click this bottom left icon here, and you click drop down, you can see like a whole bunch of options here. And these options um let you view a lot of your machine learning pipelines and training um and all that without ever leaving this notebook interface. So, a lot of developers and data scientists don't like to go to the console. They want to stay within the notebook interface. So, you can take a look at the endpoint. So, this is my endpoint. It says it's in service, uh, which means I can now invoke this model, okay? There are a lot of other capabilities here which I'll not go through. So, let's go ahead and invoke this model. So, to invoke this model, I'll just, uh, create some helper function here to load some test data from the data set and, uh, go ahead and, uh, invoke this model, okay? So, yeah. So, it picked up some random four images and the ground truth is like, uh, uh, dog, horse, horse, truck and it says all are deer. Haha. The model could be trained to be better. Let's try to run this again. So, uh, truck, bird, bird, cat, deer, deer. So, it gets deer right. So, it's It's not very good right now, but, uh, uh, yeah. So, it's 60% accurate. So, it's almost like close to chance, but I could increase the number of training epochs to get some better accuracy or I could, uh, update the, uh, or I could update the, uh, model itself, model definition itself. Uh, but this will give you an example, code example of how you can host, uh, your endpoints, okay? Model endpoints. Now, again, uh, do I see some questions here? Got to end with pictures. Awesome. Haha. Good, good. Thank you. Uh, so, yeah. Someone commented that, uh, uh, um, they're able to get to the Uh, I don't know if you all can see your each other's names, so that's why I'm not calling out names, but, uh, um, uh, I don't know if it's just me that sees these questions or if all of you see all the questions. I'm not sure. Uh, but I don't I don't want to pu- bring out names if you all can't see each other's names. So, yeah, thank you. Thank you for uh, giving confirmation that you are able to host the model. So, you can always go to the console SageMaker uh console and take a I Let me see if I can do this now. Give me a second. Uh So, here let's go back to uh studio. Am I at the right account? I hope I didn't uh kick myself out of my account. Uh Oh, this is not my account. I apologize. Uh I apologize. Give me a second. Um Yeah, I I think I kicked uh I kicked myself out of the account. Uh sorry about that. So, I'll just uh Give me a second. So, that that brings us to the end of the the first distributed training notebook. So, the second notebook, let me go through the uh second notebook here, which is I'm calling it bonus for a reason. The reason is that you can't run this right now on your You can't run this right now uh on your instances because you don't have access to the specific GPU instances that you need in order to be able to run through these notebooks, okay? So, the the example here is uh about using the SageMaker distributed data parallel back end for PyTorch, and it does few things differently. Uh by differently, it when you start to scale scale out um further, like you start with two instances CPUs, and then you start to push the boundaries of like what GPUs are capable, right? At some point, uh um using uh using back ends like these, like little more advanced techniques start to make sense. So, for small So, even for the simple example we saw, like you don't need to distribute the training for a for that small neural net on CIFAR-10 dataset, right? For CIFAR-10 in general, like there's really no benefit to doing that because you don't have a big enough dataset, you don't have like models that can really take advantage of GPUs and such, right? But uh when you are at that scale, right? If you're training large models and you want to train this on large number of GPUs, so there are few um So, I uh AWS team published like a paper on how you can uh further optimize. So, in principle, the theory is the same. You have to average gradients and you have to update the gradients, right? But there are different optimizations and different ways you can set up uh so that you get better results. So, remember I I shared earlier earlier the one of the methods was to use parameter server uh uh which was um one of the most popular approaches when you know back in TensorFlow came out like it had the only approach was parameter server that it supported and then all reduce came into picture wherein you didn't need like a central server all the participants in the distributed training could average gradients by talking to each other. So each one would get it data from gradients from its neighbor and then it would pass on to the next one right? So these were the two popular approaches. So the AWS team published science team published a paper where they have a hybrid approach where you can use parameter servers along with doing sort of an all reduce based approach sort of a hybrid approach and they found that this works better when you reach a certain scale and you get much efficient performance better than sort of the native approaches supported on PyTorch and there's more information on the documentation pages for these. So I won't go through the technical details here but the fusion the balanced fusion buffer basically it starts to accumulate gradients on your GPUs and then it it it it it partitions it into number of parameter servers you have and then shares it with them so they can do the job. So there's computation that's happening along with the GPU computation. So you don't have to wait for the GPU to compute and then do sort of the um gradient averaging it can happen in parallel and and there's there's techniques that they have built to do this. There's a paper on it if you want to dive deeper but basically it makes what you need to know is it makes sense at a certain scale. You don't have to worry about it if you're doing smaller scale jobs and because of which this approach is only supported on only supported on these large um on these large uh GPU instances. So, the eight GPU instances that are available. So, it won't run uh you can try it to run it, it won't run on run on your uh So, I I I closed my account by mistake. So, I'm going to try out try to open up a try to open up a new account and uh just walk through the notebook example and then we'll wrap up, okay? So, let me do that uh quickly here. I I did the rookie mistake of like uh just closing closing the account and I I don't remember what the account ID was. So, let me head over to studio, launch studio. Um launch app studio and uh getting started. Uh downloading workshop content. Where it is? Here it is. So, you're getting a crash course in the process I took to get to the get to the account. Um make this a little larger. We have just a couple more minutes left, so I'm not going to spend too much time on this, but your accounts will be live and running, uh which means you can you can uh uh continue to work on it as we close down the workshop, okay? So, there it is. I think the second notebook that I shared with you already has sort of the uh output on it and uh this is the second notebook. Okay, I'm not going to run this, so that's okay. So, this notebook I'm providing as a as an example that works that you can use if you were to run uh distributed training. There are a few different changes. So, because SageMaker is going to handle the distribution with its uh SageMaker's distributed data parallel back end, um you need to do a couple of changes to the estimator function like provide these distributions and mention that it's a SageMaker's back end, not the PyTorch native back end. The rest everything looks the same and the notebook is notebooks the GPUs that are supported are these. Only these eight GPU instances are supported, so it won't run on this account. But if you were to use it, then you can use these three here and then uh uh yeah, the rest of it should be similar. The core changes are different as well. So the very quick uh high-level overview of the changes is that the back end back then was glue. So here the back end that you'll use is this SM DDP. So it's a back end that SageMaker provides uh that you would use with PyTorch. Um rest of the code should look very similar to you. So there's not a whole lot of changes. You just need to use the new back end. So that's that's sort of the main reason main thing about this bonus example. Um feel free to take it to your own accounts if you have access to those GPU instances and run it. I would have wanted to give you access to GPUs, but unfortunately these uh temporary accounts don't come with these uh there's nothing I could do about it. I tried trust me. So yeah. Uh that brings us I think um to the end of the workshop duration at least. I mean, we can keep going talk about the nuances of a lot of these things. So I will also um uh share some cleanup resources. You don't need it because this is a temporary account. You're going to walk away from it and everything's going to be fine. But if you are using your personal AWS account, I think there are a few things you should know. You need to delete your end points. People are always forget this. The end points are your models that are hosted. They incur charges, right? You need to delete your endpoints, you need to delete S3 objects that you have, you need to delete um, SageMaker apps. So, there's a section here that shows you how to do this. And, uh, also, you need to, uh, also, you need to, uh, yeah, that's about it. And, yeah, feel please feel free to leave a survey. Uh, if you're seeing it at two different places, uh, I apologize. You should see some survey either on your uh, workshop platform website. Uh, I will see this directly if you leave feedback here. Uh, please I think the webinar platform will also show a feedback, uh, which is used to sort of, um, improve the program and rate. So, uh, if you have to choose, choose the webinar platform one, but if you leave both, that's great, too. So, I appreciate all sorts of feedback because that's how we keep improving the content, um, you know, whether it was um, do you need more time, do you need less time, do you need, you know, specific uh, things, how to improve, and what worked well, what worked great, so we know how to repeat what worked great, as well. And, uh, the appendix section, um, has sort of, uh, some resources to the documentations and link, uh, and then finally, um, some resources, some blog posts that I have on distributed training, but also on choosing, how to choose the right GPUs, how to run experiments, and so on. So, feel free to check them out. Uh, I put them on Medium, um, where I like to write, so the, there's some resources there, the link should also be here. And, of course, feel free to reach out to me on any of these, uh, um, uh, social media links, and check out my blog, as well. And, I will pause now. Uh, I know this is, we're at the end of the, uh, uh, webinar, uh, sorry, workshop {slash} webinar, I guess, virtual. Uh, uh, feel free to shoot questions to me later on and the accounts will keep running for at least two two hours, at least two hours. So, it's 8:30 now, 9:30, 10:30 Western time. So, at least till 10:30 you should have the accounts. So, if you want to play around with it, try different things. Uh, you feel free to continue to do that. And then it should auto terminate. And as far as the content is concerned, this website this website will go away. Um, it will stop see it will cease to exist. But, let me give you um let me give you another link that will stay almost indefinitely, um, I would say. Without committing to So, I'll let me post it at the uh Can I post in the Q&A? Okay. So, I posted a link at the Q&A right now. Oh, I'm sorry. I think I replied to someone instead of posting it in the Uh, I'm still doing that. I don't know why. Okay. Uh I actually if someone on the webinar team if you can help me post that link um, that would be great. So, it's basically a link to this website that will that will stay on for a lot longer. So, it's just a it's just hosted. It will stay on for a lot longer. You can go through the material without having to host it yourself locally. So, uh, thanks again. Thanks for attending. We have folks from uh all over the world actually and it's nice to see that. I hope you enjoyed the material and I hope you enjoyed the workshop and uh uh, again, please feel free to leave feedback and if you have questions, reach out to me and uh the accounts will be available for you to further test out your code and play around with those hyper parameters and all that. So, with that I'll say bye and thanks for attending.