Deep Learning on AWS made simple - AWS Online Tech Talks

Key Takeaways

[Music] hey everybody welcome to my tech talk i'm shashankar aws and the topic for today is deep learning on aws made simple so let's say you're a developer or data scientist working with your deep learning projects and one of the common questions we get is how do you get started with deep learning on aws there's obviously a lot of different considerations with how do you set up your development environment what kind of compute do you have access to and how do you use them how do you collaborate what kind of storage options are there how do you scale and if you need to scale how do you perform cost optimizations how do you host your models and so many other questions that we often answer because you as a practitioner of deep learning are more concerned with training your models and getting results this talk is not about educating you on how deep learning works but it is about how to sim uh how to simplify your deep learning projects on aws so with that uh let's take a look at our graphical agenda here so there are many different ways to set up your deep learning uh workflows on aws on the far left here you see the simplest setup which is you with your laptop or a workstation where you're working on prototyping and testing your models fair enough but at some point in time you're going to need more compute you're going to need more storage you're more going to need to scale and let's take a look at after that let's take a look at a very simplistic setup where you still continue to use your laptop but use compute resources on aws and then uh we'll take a look at a free option to do this i'm completely managed option to do this and a completely diy option to do this okay few different choices for you the cloud is all about choices so you can set up your development experience the way you want it get access to the right frameworks get access to the right infrastructure and again at some point you may want to scale your deep learning workflows and there are a couple of reasons why you want to scale you may want to scale to run more experiments parallelly to find out which is the best approach or you may want to speed up your training by distributing training on your cluster and oftentimes this is a challenging aspect for a deep learning practitioner who is not an expert on infrastructure and we'll take a look at how you can do this on aws and finally the whole objective of training a model is to deploy it and consume it for your customers to consume it and which is where model hosting comes into picture and as we go through this journey we'll take a look at the developer experience how does your ide look like how do you work on your projects on aws what how do you get access to optimize frameworks and what are the infrastructure considerations and we'll take a look at cost performance and ease of use as we go through this workflows sound good so let's dive deep we'll start by taking a look at a simple but a great uh an effective workflow now if you're new to aws then you know that you can get access to computer resources on aws specifically amazon ec2 gives you access to computer resources now if you're if you're already an aws expert you also know that you can get access to ec2 to get working with your projects and as a deep learning practitioner you really need three three ingredients first you need access to compute so you need access to gpus and cpus and there are a number of different gpu and cpu instances and we'll talk about you need access to storage because this is where your data set lives and this is also where you'll store your metadata and your models and such and finally you need access to software this is your machine learning frameworks deep learning frameworks like tensorflow pytorch and others that you're familiar with but optimized for the compute environment that you're using and we'll talk about that a little more and of course you need a development environment because that is how you'll interact with the infrastructure okay so one of the advantages of a simple setup like this is that it mimics or mirrors your existing laptop compute setup now let's take a look at what this looks like very quickly okay so one of the first things you will do in order to set this up is head on to the ec2 console page and when you click on ec2 instances here you can launch an instance very quickly and easily uh one of the first things you'll do when you click uh this is take a look at a variety of amis that are available to you and an am is amazon machine image and there are these machine images that are optimized for deep learning for example deep learning for pytorch and deep learning for tensorflow and others okay you can select this and go through this process of launching an ec2 instance now i'll not go through this process because it's fairly easy to do and once you've launched an ec2 instance you have an ip address which you can then use to connect ssh into it okay you must start to notice now that this is not an ideal developer experience but nevertheless you get access to an ec2 instance and in this case you have access to a gpu instance uh i can conf confirm this by running nvidia smi i'll see here that i have a nvidia v100 gpu right and you can also launch a jupyter server like i've already done here um just pull up a jupyter server and you have an additional step of setting up a tunnel so that you can access your jupyter server from your local laptop okay uh you must notice now that this is this is a pretty pretty involved workflow right and you can pull it up this way so go to localhost 8888888 and lab and you have your id so after all this setup you get access to this nice jupyter uh client on your end where the server is running on ec2 instance and you can let's go ahead and run this notebook it's a simple tensorflow notebook we're just training on a cipher 10 data set and we can pull up uh um nvidia smi and take a look at uh gpu utilization as you train okay so now we we're going through the process of uh training here and in a moment you will see here that you will see gpu utilization spike up there you go so the the benefit of this sort of a setup is that the benefit of this sort of setup is that here's your gpu utilization around 60 or 90 percent uh it's not a very complex model it's a fairly simple model but the benefit of this setup as i mentioned is that you get access to um a developmental environment that you're familiar with that's hosted on aws you get access to up to eight gpus on a single ec2 instance and also up to 16 terabytes of local storage on ebs right amazon yeah ebs and uh you get access to the latest frameworks uh i briefly mentioned this while choosing the ami uh here's a documentation page for all the deep learning amis you get access to a variety of deep learning ami that's optimized for deep learning and they're designed with support for gpus and cpus x86 as well as as well as graviton uh they come with the cuda for gpu support as i mentioned and there are a range of these environments that include several different conda environments for tensorflow pytorch and others okay these um and you'll also see this um available here so you have conda environments that you can choose in order to get so this simple yet effective setup is um it's a great way to get started and there are many different gpu options on ec2 instances i'll just leave a resource here this blog post i go through several different gpu recommendations okay all right so a couple of tips here the first thing is uh i highly recommend you use the frameworks provided by the deep learning ami and don't tip install or conda install deep learning from frameworks from upstream simply because the frameworks that are available upstream are not optimized with your hardware in mind um remember these upstream con frameworks are designed to work on your laptop or on the cloud or anywhere else you're gonna run it so it doesn't have the specific optimizations of easy to always use the frameworks provided by dlami because they've been tested and optimized for cpu and gpu instances on aws okay so the challenges with this approach is that the id setup is complex and it's difficult to recreate and reproduce this environment if you're going to share this with your colleague or or someone else and of course it's not easy to scale of course you can get to eight gpus on a single ec2 instance but you can't scale beyond that easily without managing networking and orchestration and all that okay so what if you wanted a fully managed id experience now there are two options on aws here there's a free options yes you heard me right i said free option which is sage maker studio lab ide and then the sage maker studio ide which is a paid option but gives you a more complete deep learning end-to-end experience with several other capabilities that we'll talk about let's start with the free option that's available to you now if you head over to studiolab.sagemaker.aws you're greeted with this home page here you can request an account i already have an account again remember this is a free option and you log in and you get access to cpu and gpu compute resources both are completely free i'm going to start up a cpu resource and once it gets started you get access to a hosted jupyter notebook interface and up to uh 15 gb of persistent storage so between launch between sessions you all your data is stored and completely available to you and you click open project and you get access to your sage maker studio lab uh i'm i'll emphasize this again this is completely free okay you can get access to your own training and prototyping environment you can train your models here okay this is your personal playground if you will and you also have access to gpu compute resources the gpu compute resources will give you four hours of training time and cpu will give you 12 hours of training prototyping and testing time great to learn and get started okay that's studio lab for you now the other option is sagemaker studio id which is which is for your production workflows okay with stagemaker studio ide what you get access to is let's head over to the sagemaker console click on sage maker studio so we already saw studio lab here okay which is the free option now i'm going to click on studio and launch stage maker studio and i already have a studio domain setup uh it's easy to go through set up your first domain and then launch up my launch of my studio ide okay so what i get access to here is my studio ide with a completely hosted uh fully managed jupiter notebook experience here okay so here are a couple of notebooks i have we'll go through some of these examples uh one of the benefits of this is you can switch compute back and very easily uh take a look at all the different compute back-ends that are available to you so there's no instances to manage there's no instances to launch just select the computer resources you need and switch your end and start training okay and studio also comes with a host of other capabilities that we'll talk about such as experiment management monitoring your endpoints you know compilation jobs um you can monitor your endpoints right here hosted um several different capabilities that we'll talk about so let's head back to our slide here so you can set up your own jupyter-based um environment on ec2 or you can use one of these managed options uh in order to do your projects now a quick tip here use the free studio lab for learning and prototyping create an account takes a couple of days to get get access and then you can use it but if you're doing anything for for production such as you need access to experiment tracking collaboration scaling hosting models usage maker studio you get end-to-end uh cicd options for ml ops with uh you know sales maker pipelines you get better options with managed spot training to save cost and all that whole host of capabilities okay now the benefits of this setup of course is that compared to the easy to setup is you get a fully managed uh jupiter-based experience and uh you get access to again aws optimized ml frameworks uh with deep learning container environment so when using sagemaker studio your environment is basically a containerized environment if you don't know what that is uh hold on i'm gonna cover that and finally you get access to single you get cpu and gpu options with studio lab which is free and with sagemaker studio you get a whole range of compute options as i showed you so that you can choose the one that's right for you if you're doing development prototyping on this instance okay so though so far we covered simple setups right how to take your deep learning to aws and be productive with the right development experience framework and computer options but often there is a need to scale and there are a few different reasons you want to scale you want to scale either to manage multiple experiments or to speed up your training so let's talk about that a little bit now one of the challenges with scaling or using more compute resources is that there's an assumption that your code which is working on your development system either your laptop or on ec2 or any specific development environment that you set up that we've discussed so far and you take it to another environment which is your training cluster or that you self-manage or a managed cluster and you hope that the code works but in reality the ml stack is pretty complex which means you may be using a different version of framework a different version of framework is running on on the cluster there or there's a mismatch in the gpu libraries there's a mismatch in the driver and operating system and so on and this introduces multiple points of failure okay and this is the biggest headache customers face with migrating machine learning code and it's exceptionally more challenging than a traditional software just because of the nature and the complexity of an ml stack which has tighter integration with hardware basically your cpu and gpu libraries and drivers so one of the solutions to this is to use containers now containers give encapsulate not just your training code but also all your dependencies and configurations into a simple lightweight package that's portable and scalable now containers have their roots in traditional deployments not machine learning but very quickly the machine learning community has adopted containers not just for hosting models but also for development workflows as we'll see now uh they give you this portable development environment that you can share with others and hope to reproduce results regardless of where you're running it now if you're a seasoned deep learning practitioner uh um who's probably new to containers a question you may have is like hey i use conda environments i already make sure that my environment is portable and reproducible but remember that conda environments only encapsulate uh your code and your frameworks and at most python versions but they don't really encapsulate your cpu libraries optimize for hardware such as your mkl and others your gpu libraries which include your cu dnn and kublas and various other gpu libraries that deep learning frameworks use and also drivers and also don't encapsulate uh the os and os kernel and all that okay containers on the other hand um not just encapsulate your code framework and dependencies in python but will also encapsulate your cpu libraries your gpu libraries and drivers can also be containerized into their own dedicated container driver containers so that what you get is a an even more portable way of sharing and reproducing results that is not possible with python environments or other mechanisms there are virtual machines but they're too big and too cumbersome to share like sharing an entire ami they're too big and too cumbersome they're not lightweight they're not portable and containers strike this nice little balance between portability and lightweightness for your machine learning environments which is why they've become so popular uh for machine learning workflows okay so that's the summary of why you would want to use containers now oftentimes as a deep learning practitioner you don't want to deal with infrastructure and containers and i completely empathize with you i don't want to deal with containers either which is why a lot of these optimized containers are readily available to you aws makes them available on on elastic container registry which is where container images are hosted now if you head over to this link you should see a big list of containers options here now there are several different containers let's take a look at let's take a look at some container images in this repository under available images so you'll see container images available for let's say tensorflow let's go to tensorflow here and see a list of tensorflow containers so this uh python containers tensorflow containers for cpus and gpus and optimize for distributed training with hardware yes or no for training and for inference and for different regions okay these are again optimized frameworks that you should be using rather than using upstream frameworks because these are optimized and tested for aws hardware okay i highly encourage you to use this instead of using your own if you want to customize a container or by all means just pull one of these container images and customize it if you have the experience to do that or you can start with one of these container images uh now you're probably wondering uh okay there's a container image but i don't know how to use it right fair enough that's a good point so we'll discuss how you can take advantage of this uh leverage these for your workflows okay heading back to our slide here one of the benefits of having everything integrated is that you get better performance by integrated i mean having the compute layer which is your cpus and gpus and hardware storage options all the networking and frameworks that you're going to use with containers working together and one of the services that enable you to do that is amazon stage maker because with sagemaker you get a fully managed infrastructure and training service uh which way you just bring in your training scripts depending on um whichever framework you're working with and it had enables you to uh scale and you ensure that you're getting the best performance of your computer resources on aws so there are a couple of reasons to scale okay why i keep talking about scaling everyone keeps talking about scaling deep learning right and oftentimes you want you may be wondering i i don't have a need to scale why do i need to scale so one of the reasons why you want to scale is to run multiple experiments now uh many of these experiments could be finding the best hyper parameter finding the best model architecture or maybe simply training on different data sets and you have to go through many different experiment steps and these are all embarrassingly parallel which means you can run all of this on different compute instances now if you were to run it on the same instance in a loop versus running them in parallel on multiple instances the cost to you would be the same simply because you're using the same resources but in parallel versus in a sequential manner right so there's a need to scale out experiments so that you can get faster results but as a deep learning practitioner i understand you may not have the infrastructure experience to manage a cluster like this which is where sagemaker fills in that gap and we'll take a look at an example okay so let's head to our code example here so back in to my stage maker studio so one of the best ways to take a look at our experiment management is let me open up an ex example here um here's a jupiter notebook where i have an example of running a parallel experiment job in multiple jobs and taking a look at results to find the best combination of hyper parameters now this example is based on this blog post that i have here which will be linked to you um and i'll share this to you at the end of the presentation so the example uh here goes through the process of creating an experiment and an experiment is um something that you want to achieve you want to decide if a custom model architecture delivers better results than say resonate or vgg or something like that so you can define what it is so you can create an experiment in sagemaker and then you can specify a combination of parameters that you want to run in parallel so maybe you want to run resnet and custom architecture with different optimizers and different number of epochs and all that right you can enumerate these and in sagemaker you can create an experiment and you can loop through it and run through all these different combinations the benefit is that sagemaker will run all of these in parallel so each of these experiment trials will consume one gpu uh one instance and they're going to run in parallel okay they're going to run in parallel and at the end of it you can take a summary look at all the different um results now after running all of these i can sort by test accuracy and decide which gave me like the best was uh my custom model architecture with adam optimizer and for 120 epochs versus the worst was my custom architecture with sgd you know right here at the bottom okay so the benefit is that now i ran all of these in parallel and i can take a look at the results and i notice i never needed to set up i never needed to set up my infrastructure so all the magic happens here at the sagemaker estimator function where i simply loop through running multiple of these and these are fully managed as an experiment now sagemaker studio gives you access to managing these experiments right here within the jupyter interface so you can go open this up and take a look at all the training jobs that we ran here these are all dedicated training jobs that ran to run your experiments okay so this is a simple example even if you're not distribute training to take advantage of multiple resources now the second reason you want to scale your deep learning projects on aws in a simple way is to get faster results and how do you get faster results there are two strategies the first one is called scaling up and simply what that means is if you're running this on a cpu instance or a smaller gpu instance just get a bigger instance type with more compute power and that is called scaling up so replacing what you have with something even more powerful is called scaling up and this will be the simplest way to do it because this requires no code modification in most cases uh you simply if you're using a um you know p2 instance try a p3 instance and if using p3 instance try a p4 instance and so on and so forth so just try a more powerful gpu instance the other way to speed up training is um you maybe you have large data sets and you can actually scale out training scale out training is distributing your training across multiple instances so you could go from tens hundreds or thousands of gpu accelerated instances on aws and scale out your training to get results faster the downside of scaling out is that you need cold modifications as we'll take a look at an example you need to modify your training script in order to be able to scale out uh just because um of how scaling out distributed training works where you need to average the gradients across these different instances again the only effort on your end is to modify your code you don't need to manage infrastructure because sagemaker will do it for you now some tips again here always always always always scale up before you decide to scale up scale out what i mean by that is try to find a more powerful instance that may satisfy your time to train goals before you decide to distribute a training because distributing training brings with it its own challenges by having to modify your hyper parameters and all that okay so don't distribute training before you tried a more powerful compute instance and there are many many different gpu instance types on aws and uh um you can you can take a look at this i have a blog post i'll link to it at the end of it recently updated with all the latest gpu instances so i have key recommendations on which gpu instance to get started with the highest most highest performing instance to highest performing multi gpu and single instances best bang for buck instances um best performance for cost ratio and all that and at the very end of the blog post i also include a summary of all the different instances that are available on um aws for you to take a look um here so i have a table with all the different capabilities so don't uh jump into digital training before you exhausted scaling up or adding the most powerful instances okay that's my pro tip for this section now if you do want to do distribute training there are a few different options um sagemaker makes it really simple for you and the whole key idea between distribute training with distributor training is that you can take your data set um rather than process it sequentially as you do in a single instance you can distribute it to multiple workers but then there's a step of averaging gradients and updating all your workers which add some overhead okay now um so effectively you're speeding up the batch size so you can speed up the training now let's take a look at an example of how this is uh this can be achieved simply on aws okay training with a single gpu is pretty easy with sagemaker the whole key idea behind this is if this is your development environment by this i mean this jupiter environment is your development environment then you can offload the training to a dedicated training instance rather than training here and the whole idea is that you can decouple your development environment from your computer environment and behind the scenes what happens is that you will bring in your training script and sagemaker will pull in the appropriate deep learning container whether it's tensorflow or pytorch combine them as in load your training into this container image spin up a cluster of one two or hundred or thousand gpus you want and start training download the data sets from s3 automatically so here's how it works this is this is at a higher level and again the key idea is decoupling your development environment with your training environment so the uh key idea is very similar you specify your hyper parameters and you set up an estimator function now there are a few important things here you'll specify that you want a gpu instance you want only one of it you can specify your tensorflow framework version if you want um you know and then your python version and then you call the fit function to start training now all you need to do is bring in your training script here which is in my case cypher 10 and let's go take a look at what this looks like so in my code example here and this is your very simple tensorflow training script nothing fancy here we're processing our data sets and then you know downloading our resnet50 model um or your custom model here and then going through the training process uh nothing fancy here this is your straightforward training function here that you would you know run locally but you'll add some parameter parsing which tells sagemaker that it can pass your hyper parameters as command line arguments so that it can train on these okay so after you after you do that training you call the schmidt fit function and sagemaker will spin up this instance for you automatically and go through the process of training on that instance and at the end it will give you training seconds and billable seconds the benefit of this is of course you don't need to constantly keep running a gpu instance in order to uh train you can run your development on a just a cpu back end like as i'm doing here i'm just running this on a cpu cpu development environment it costs less and then i can still train on a gpu instance because my development environment is decoupled with my training environment and of course i can take a look at my test results too with my test accuracy and all that so uh yeah that is training on a single gpu so what does it take to scale to uh distribute training almost no code changes as far as running with sagemaker is concerned some code changes to your training script okay so the only code change you will make to the estimator function is you'll add a new distributions parameters so you'll call it sagemaker distributed data parallel mode enable true okay this is important because you're telling sagemaker that it needs to launch multiple instances and instance count here is in our case um uh one instance count but it has eight gpus eight nvidia v100 so you want it distributed across eight gpus now this could be ma 16 or 32 whatever you want right and the code is a slightly different code because it requires modifications so the rest of this script is trained you know estimator function and fit but you need to modify your training script in order to be able to uh take advantage of distributed and this you would have to do this no matter what no matter which approach you are using to dissuade training the infrastructure is managed by sagemaker but you have to make modifications to your training script so the few modifications include uh communicating the size of the cluster in this case eight gpus um and then you will want to make some updates to uh i'm using gradient tape here you need to wrap the gradient tape with the with the distributed gradient tape api that's provided to you by sagemaker distributed and few other changes to broadcast um you know the variables the gradients so that you can do an all reduce operation and the all radius operation is here um again you most people don't need this i don't expect you to be using distributed training most people don't need this but if you do i have a blog post where i spend more time explaining how this works again my my tip for the day is always scale up always scale up before you scale out because scaling out is more complicated and it introduces more opportunities for errors okay so another question we commonly get with deep learning projects on aws is like what are the best storage options uh if you're modded to large data sets i highly recommend using amazon s3 because with sagemaker sagemaker will automatically copy your datasets from s3 to the training cluster and if your data set is large it can also stream your data set from s3 to the training um instance where you're where you're running the training now if you have uh data sets in a shared file system like amazon efs that's perhaps being shared with other applications or other use cases then you can also use uh mount such a file system to the training cluster so that the data set is accessible to all the training instances whether it's one or multiple and finally if you really want the best performing state-of-the-art uh file system then fsx for aster is also supported which behaves like a file system but it's also natively integrated with amazon s3 which means all the you get the benefits of a file system access uh but also backed up into amazon s3 so these are your options but s3 will satisfy most deep learning use case needs okay so we've uh we've covered um you know different we've covered uh different development environment approaches free as well as fully managed and diy with ec2 we covered scaling for experimentation as well as distributed training and how to simplify that with amazon sagemaker now let's take a look at model hosting and deployment and this is often a challenge for developers for data scientists specifically because they don't want to deal with hosted models and sagemaker really simplifies that with with a couple of clicks and the benefit of that is that you don't need to rely on an infrastructure expert to do this you can manage it yourself and let's take a look at a few different options so there are several different deployment options uh the two we'll discuss today is sagemaker real-time inference wherein you have a dedicated instance that you can use to serve inference requests such as a cpu instance or a gp instance but the more exciting one in my opinion is sage maker serverless influence wherein you don't need to manage infrastructure at all uh it's it's based on invocation so you you'll you'll only be charged based on invocations um you don't need to manage the instance type especially if you have sporadic requests coming in through the day then this is the best option it recently became generally available and we'll go through a demo of this the few other options such as asynchronous in inference where you can um request a queue if you have multiple requests coming in throughout the day you can have it queued and processed asynchronously which way you are not waiting for a real-time response or even a delayed real-time response you you basically don't care about the response time uh you can save some cost by doing asynchronous influence and finally there's batch transform if you're doing offline processing of large large large batches of data so we'll take an example of the real-time and serverless option here okay and let's let's go to our code example again um a lot of i'm going through a lot of demos but i'm going to be sharing all of these with you these are all available on my github i'll share a link with you at the end of the talk and let's go through the high level overview of how deployment works on aws and share a code example with you okay at a very high level um deployment involves three steps so once you finish training your model on sagemaker whether it's locally on an ec2 instance on sagebreaker studio lab or sagemaker studio or on distributed training with sagemaker training whatever it is you create a trained model now this model can be needs to be registered so you need to create a model uh using a create model api and in this you will specify what kind of serving container you need if it's a tensorflow model then you'll use tensorflow serving if it's by torch you can use tor serve there are also other serving containers such as nvidia stripe and server so you have several different options and in the code example i'll go through this after you create a model you'll create an endpoint configuration and this is where you specify if you want a serverless endpoint or you you want a real time endpoint as i mentioned serverless is uh when your sporadic request uh serverless you don't have to manage uh resources it will automatically do this for you with real time you have to specify the instance type you want a cpu or a gpu and so on and so forth and then you can create an end point it's three steps okay now uh i go through a lot of this in this video um that that's linked here that i'll share with you but essentially uh essentially i will go through um a brief uh overview of what this looks like uh in this image here so with serverless as i mentioned if you if you're a say a retailer you get you get traffic on specific times in the day for example in the mornings and in the evenings but you have no traffic during you have no traffic during the middle of the day right so having an ec2 instance or or instance continuously running will cost you the cost per inference will be higher simply because it's running all the time whereas if you have this kind of sporadic intermittent request it's better to have a serverless inference model where you're only charged for invocation and not for the duration of the instance running the downside of this is of course you don't get to choose a compute type and if the endpoint has been idle for a while there's a cold start period so there's a cold start period that will come into picture if if we if it's been idle for a while if you do have uh traffic that's continuous throughout the day with sufficient number of requests coming in then it may make sense to have a dedicated instance with real-time inference so these are your two options and i go through both of these in in this video in more detail so i encourage you to check it out so serverless for intermittent traffic and real time if you want that high performance low latency performance with continuous traffic throughout the throughout the day so let's go through this example to see how each of these options work and remember we'll go through the three steps of creating a model creating an endpoint configuration and then creating your endpoint okay so the one of the first things i'll do here is actually download a trained model and for the purposes of demonstration i'm just going to download a resnet50 model and then i am going to package it up and upload it into amazon s3 here as you'll see here session.upload to amazon s3 okay so you can make your model available in s3 and it doesn't matter where you got the model from it could be from your training job or it could be something someone gave it to you and then yeah you want to choose your model serving container and there are several different options in this serving container list here you can just go search by inference for pytorch and tensorflow and you can pick the one if you don't want to pick the if you don't know how to pick the right one then just use this api here to retrieve the right framework for you so you can specify uh tensorflow region the version python inference and so on and you can get access to the container image that you're going to use to host a model and then you can create a model using i'm using the boto3 api here and you'll specify uh that you want to register this model with this specific container image and then after that you'll create an endpoint configuration where you can specify that you need a serverless endpoint okay serverless endpoints won't let you choose the compute type because they are supposed to be serverless you don't need to manage infrastructure but you can specify the memory size how much memory you need depending on the model size also how many max concurrent requests it can handle it can range from 1 to 200 and after that you can create an end point creating endpoint is done by using create endpoint api and then it goes through the process of creating the endpoint and then you can make a request so very first time you make a request the wall time is a little the the invocation time is a little high because remember i mentioned that there's a cold start period the so whenever the end point has been idle uh there's a cold start period and once the end point is warm as in it's already been executed the next subsequent time you run it will be much shorter and this is true again subsequent request will be quicker and you you don't get any requests for like 10 min for for a long time and then you there's a core start period again okay and finally uh you can switch from a serverless endpoint to a real-time endpoint let's say your traffic is going to be more predictable moving forward then you can switch by creating a new endpoint configuration where rather than serverless you say it's a dedicated instance which is a real-time instance and then you can call an endpoint update option it will just update the serverless endpoint to a real-time endpoint and then you can invoke the real point in point uh of course you can choose now between cpu and gpu options and so on okay so that's a quick demo of how to deploy your models without managing infrastructure on aws okay so that brings us to the end of uh towards the end of this webinar now i want to share some resources with you first is i have a number of blog posts here that cover how to choose the right gpus how to choose the right inference accelerators how to manage experiments how to do distributed training debugging models and so on feel free to check them out it's on this blog post here blog here and also all the code for today's webinar is available on this url deep learning on aws feel free to check them out as well if you want to replicate some of these results and i thank you again for taking time to listen to me today my contact information is on screen feel free to reach out to me any of using any of these channels and i'll see you in the next tech talk thank you [Music] you