Workshop: Going Production: Deploying, Scaling & Monitoring Hugging Face Transformer models

hi everyone hello [Music] hello we are just waiting a few more minutes until we get started [Music] [Music] to the ones who are already like in the chat and so who of you has attended last week's workshop where we like fine-tuned our model cool a lot of new people awesome so it's like it doesn't matter that you haven't been um there last week so we are not like building on something we have created last week it's like completely something from scratch so um we can we can put the discord link because we are accepting the questions from there um let me post it okay great i think we can start with the introduction and then for the folks who's like joining a bit late it doesn't matter since we are just giving like a bit insights on who we are and what we are going to do so welcome from our side to our second workshop and going production where we are deploying and scaling monitoring i mean first transformer models on aws h maker as well as running some batch transform shop and today joining me is nate who will take care of all of your questions which are going to be asked on discord or in the chat um she will do the session about batch transform and then myself and mathieu from aws so welcome to our our workshop and here again the quick links um mervy has already shared the discord link we also have a very active community forum on discuss huggingface.co with a separate section for sagemaker so if you have any questions about hugging face on sagemaker after the workshop or like in a few months feel free to open a thread there and all of the code which we are going to show today are is available on github and on the repository shared there i guess but you can take over and give a short introduction yep we will do a first introduction for sagemaker so for the people that was there uh at the previous session it will be a bit of repeat but i will go faster for you but if you didn't uh join the first session i think it could be useful so let's go for that so the adobe's vision uh for the machine learning it's to make the machine learning accessible to all the customer they if you are developer data scientist or business user uh the uh we sign a very strong partnership with aginface to make the nlp easy and accessible for all and as you know like hugging face is the most popular open source company that provides a state-of-the-art nlp and on the other hand you have the sagemaker offer from aws that give you a high performance resource for training your models the iws ml stack is composed of three layers so at the bottom layer you have all the infrastructure parts where you will provide like a gpu but also custom chip for the inference and and also docker container uh especially built for the deep learning on the the upper stack the ii services is where you are only developer and you want to put like more intelligence on your website and we have a bunch of services that help you to do translate sentiment analysis but also focus recognition for images and on the middle stack that is like the most important for you is sage maker and sage maker is a tool to help data scientists to go from the id to the production that will be a big focus today and so we provide a lot of tools that help you going from a notebook to a production endpoint where you can make the inference and on that stack of sagemaker we include again face as a custom as a framework that is fully supported by aging phase with what we call the dlc the deep learning container and so uh we gonna go through the introduction of the again phase deep learning container so he help you to to train and do the inference and whatever frameworks you are using if it spiked or should sensor probably will work we have an estimator that is directly integrated to the sagemaker sdk so it's really easy to use and uh and you can start a cost effective job for training without worrying about docker we have a comprehensive gallery of example and everything is support and maintained by adwords so if you already have a super substitution for your duplicate seconds you will benefit from uh from adware's support even for again phase training job and inference so the benefit of running again face on site maker will be cost effectively because you can leverage some container that is deeply optimized for again phase training we have spot instance if you want to leverage those for a non-critical workload we we follow the envelopes strategy so you can automate everything inside sagemaker from the meta store the logistics inside cloud watch you have the monitoring and the profiler and of course experiment it's really scalable so you can spin up any gpus that you need and to do parallelism but you will also can can downscale if you don't have a maybe user during the night or something and of course everything that we do at aws security is like job zero for us so you can leverage the vpc connectivity and the fine-grained ein permission for securing your workload so again face again on sagemaker is like a really a top quality container that we develop with with again face and it will remove the heavy lifting of of managing a nicer registry i have custom installation and sdk so we will really streamline the process uh for using your interface on sagemaker so the way that it works is like you build your model by leveraging sagemaker notebook but it could also be the the studio or your own ide that you use on your yoko machine you will start a training using the dlc and that dlc will uh will pull uh the the models and all the frameworks from again face you will uh from there fine-tune uh your model and manage your experiment and after when you have done and that you have your model that is trained you can uh deploy it using sagemaker endpoints or and automatically leverage the scaling of that model and we will have a deeper dive today on that or you can download it directly from s3 and host it yourself so now uh we can get your hand dirty i forgot to update the the ash code so it's the wrong one but you will have it uh philip just share it in the chat and so you go to that link that is the dashboard event engine so you will land on this page you put the edge code just there and that will uh with that you will be able to access one adobe seconds so for the signing methods uh you can choose the the email one-time password so from there you can type your email you will receive an email with the one-time password waiting for again and you land on the the team dashboard from there you can click the address console button and then click open adobe's console from here you access your temporary icons inside adws and on the top bar you can search for sage maker [Music] and go to the search maker service page here we already spin up some notebook that you can access so you click a notebook and then a notebook instance and here you have four notebooks so you can pick one the only thing is like you are four by accounts so when you choose one you will need to take care that someone else didn't took it before so you click open jupiter and that will already accu to the jupiter notebook for that and from there you can click on running here to see if someone else uh have already start a notebook if it's not the case you can go to the second workshop the lab one for for the first demonstration and stop the notebook and here you have the first notebook and not that if i come back here and i refresh there okay you have the green book that say that is already running so if it's the case and you see the green book or uh the the notebook there you just go back to the searchmaker console and you pick another notebook and so i will end up tonight to start digging dive deep into that notebook actually we're going over to uh mirva today oh marvin yeah sorry for that the floor is yours no problem maybe before we get started you guys just see like a few questions coming in the channel to access and are the hash working for you or is there something like um say saying what errors i saw that toby has an error with throttling exception right exceeded and also michael uh um don't know how to come to the console okay michael managed to works okay it works hash works works perfect okay and devin when you're like a team dashboard uh you can click on open console and then go to uh search for sagemaker at the search bar and then you should come in as the sagemaker notebook yes just on top here should i start yes this is yours okay let me show my skin okay you can see my screen right yes um so today uh last week we have uh we have trained the model successfully and deployed it and this week we are going to do a batch inference with batch transform so our problem is our data consists of tweets and these tweets have sentiments and we are going to classify the sentiments of those tweets um there is a i'm going to start with pre-processing and then tell you how to set up the transformer so firstly you start with uh starting the sagemaker session and then get the exec get execution roll for the user roles and set up your s3 bucket and we have this tweet data we are going to turn it into a json lines file uh here uh it's just we are transforming it into a different format and we are going to upload it to uh to s3 with s3 uploader we are just creating this bucket and in this bucket we have this batch transform input folder and batch transform output folder and our tweets look like this so if you have previously trained the model with a hugging face estimator already you can directly uh create the transformer for that estimator uh you have just trained your model with and if that is that is actually your model sorry and then call the transform on it you can still use the use the models in the hub so let us go through the notebook so i'm going to i'm going to i have just uploaded uh transformed the csv file and uploaded to the s3 bucket and i'm going to create this uh transformer to run the bat job from uh but for this time i'm going to take the model from the hub so i'm going to have this environmental variable uh hugging face model id this is the this is our model and we are going to give the task which is text classification here and we are going to create the hugging face model so we give the environmental variables roles uh transformers version pytorch version and python version uh we create the transformer uh that we will call later call our batch uh transform a bit so i am giving uh instance count instance type output paths which i have just created above and the strategy which is single record meaning that we will we will send one uh one example in every request so i'm going to run this and to uh do the inference in a data set because in the in the in the batch transform you pass a data set you do this in batches uh to pass the data set i'm going to take the output file and um i'm going to i'm going to download the output file and then uh i'm sorry uh i'm going i'm going to get my uh i'm going to get my results by doing uh by downloading uh the the the result of the batch transform which was in the output path so i have downloaded it with s3 and i got my inferences which is basically uh the label one is positive label zero is negative these are the confidences so yeah philip would you like to explain this yeah sure thanks so um when you like so we we are doing two since lab one would be basically just like uh creating a real-time endpoint which nate has done in the last section but feel free to do like one if you are interested in like after the workshop you will have um access to the um temporary account i think from 60 more minutes after the workshop is done and so you can run lab one as well and when running lap two when you execute a batch transform job um it will take a few min minutes until like the instance um is spawned up and then run the batch job and what we can see here on the visualization is basically an explanation of how um aws stage make a batch transform is working behind the scene so we have a real-time endpoint which you can like easily deploy to sagemaker and then send http requests back and forth but um for real-time endpoints the instance is basically up 24 7 so you would pay um the duration of the instance time being up and that's basically nice if you have like real-time application but a lot of enterprises are still in the process of hanging batch shops where you run like predictions overnight or like once a week to like update your database for example with a new model and that's where batch transform can chime in and batch transform basically works um as we see here we have like one instance in this case and the instance consists out of an agent and which is basically a python script um ec set and the model the model is a real-time endpoint we know from just deploying a real-time endpoint and the agent is basically the plane of our batch transform job it will load our uploaded json line data set from s3 and then iterates through it like so we have like a json line data file with maybe hundreds of thousands of um inputs and the agent will iterate through it and send one http request per row to our model endpoint and saves the output or the prediction of this line into a new json line file which will be at the end uploaded to s3 so the whole process is basically you provide a data file a json line data file and start a batch transform job the agent will iterate over it sends the request to the model endpoint and creates a new output json line file which will be uploaded to s3 and then sagemaker will take care of stopping everything at the end and it's also possible um to scale up to multiple instances so if you are like not running like on a few hundred or a few thousand inputs you could easily scale up your bets transform top to two four or even eight instances and then like the data will be spread evenly for multiple files and then it's way faster to iterate and you will only pay for the duration of your batch transform job and if you check your locks and when after the execution of batch transform job you can see that it's like simple http request sending to our model endpoint okay um yeah thank you uh we have a couple of questions yeah maybe to ask amir khan's question so what about the tokenizer so um as you can see if not scrolls up a bit where we define our hub configuration with a model id and and the task and this will also load the tokenizer from hugging from from the hugging face-up and then our inference dlc load the tokenizer loads the model and then runs the prediction by itself i think it's around here that i have loaded the pipeline yeah in here and you give the when you give the task it also loads the it and the model id it also loads the tokenizer um and then to answer hanos question how do i use the model not in the hub so instead of providing when creating the hugging phase model class from the sagemaker sdk currently we defined the nth parameter with the hub configuration and instead of providing the nth parameter you can provide a model data which is an s3 ui where your artifact is stored you can also um go to the github of hugging phase with called notebooks there's like a there's like a whole stage maker section um with examples for batch transform jobs or deploying two um sagemaker to real real-time endpoints using s3 artifacts instead of a hugging phase model hub model also in the beginning of the notebook there is an example of uh how you train from scratch i mean it's it's not how you train from scratch but if you have trained from scratch and not from not have you haven't taken your model from the hub how you can perform this is given in the beginning of the notebook okay can we have like a message in the chat for those who started their batch transform job already how hard jobs going and if you have any questions at all just feel free to drop them in the chat here guys perfect there we have like our first finish successfully that's amazing and maybe to like to explain a bit more why we are like converting the csv file to a json line file it's basically um csv is just like um column based and since we are working with text it's like oftentimes you have for example the delimiter of your columns in a csv inside your text so attacks can easily contain like a colon or like some special characters and then this could like basically screw the the data processing on the inference side that's why we went with json9 where we have our dedicated inputs and also um additional parameters so the batch transform job is not only working with text classification for example you could also do like zero shot classification and then you couldn't even like represent it properly inside a csv since we are having like the inputs and then our like candidates for our label which is also an array and working with like nested areas inside json is not very nice and maybe there's a few questions is there any limit of fetch i mean what do you mean with a limit limit in time and like fire size limit or a duration limit or no there's no limit like if your json line file has instead of like the few hundred rows in our example like a few thousands or few hundred thousand rows then the batch transform shop is just running longer that's all and then maybe for vince news question and yeah of course you can like use q4 pipelines but i think you would need to like create your own container and like scripts to work with it i'm not sure if you can like easily use the shmaker dlcs for that inside of cubeflow so that's something you would need to try out okay then i would start sharing my screen and we would go to let free which is like the the lab where we are going to scale our yeah inference and monitor it so basically let three is similar to lab one where you create in real time endpoint except that in lab three we are going to add additional configuration to auto scale our endpoint and to directly look into the um monitoring dashboards created by cloudwatch and be free to ask like all of the questions we will have some slots when going through the example where we can answer them and you can find the the notebook again in workshop two and then it's like three um auto scaling and for this example make sure we have like a higher stage maker version than uh 2.6 installed because we released the new dlc version supporting like the latest transformers version so i have done this already just to check again i have like a nice higher version and then what we are here doing is basically creating a real-time endpoint so we again pick up our im role have we we will use again a hugging face hub model in this case we are using a fin bird model which is nice because i guess it's more real world oriented it's fine tune on a lot of like financial text corpus to estimate the sentiment that's why we go with this model you can find it on the hub and then our task is again text classification for the name and i will use the second part of our hacking phase id and then just a uuid to make sure that we are like creating all of the models and then we are again creating our hackingfest model class with our hub configuration and in this case if you like already have a model on s3 you could replace the end with the model data pointing to the s3 ui we provide our role the endpoint name and we are using transformers four point eleven pi charge 1.9 and python 3.8 and then we are creating our endpoint with the dot deploy method to a c5 instance which is a compute optimized instance with an intel cascade lake and we are also defining one variable we need later for accessing our cloud watch and dashboards directly from the notebook so it's just the aws region where your endpoint is being deployed and then for those who don't know basically what's happening behind the scenes when creating a real-time endpoint so here you can see the like the software structure of our inference dlc the inference dlc is the hosting service where we use the multi model server which is an aws created machine learning serving um yeah system based on java on the neti framework which provides the i and also when you are sending a request to the elc we have these emails this mms interface written in java which will accept the request and also respond the request but then after nettie has received the request it starts python worker so the prediction and the inference code is written in python but the whole wrapper the i o wrapper around it is based on the multi-modal server and neti and the inference toolkit is an open source um implementation you can also find it on github and it's based on the hugging phase transformer pipeline so basically it's an optimized wrapper for stage maker for the hacking phase pipeline so for example if you have like a project with the client and want to showcase a model you can load it with the hugging face pipeline and then if you want to like create an end point you can basically provide the hugging face id and the task and then you will get a similar interface then you have with the pipelines and the inference toolkit also supports all of the quarks so if you are working with a generalization tag like a task like summarization or translation you can provide like the additional arguments as parameters for like um penalties or max length or min flanks these can all directly be provided inside the uh request and you can find more or basically all inference parameter on the acme phase dot co slash doc stage maker side there's a separate section for inference and then what the inference toolkit is basically doing it's idle loading the model from the hugging face hub or from s3 initializing and then depending on if it's a tensorflow model it uses tensorflow if it's a pi touch model it uses python pytorch and then initializes the pipeline and um serves the the request so when a request comes in mms accepts it make sure it's correct passes into the inference toolkit which runs a pipeline which passes the incoming inputs and parameters to make sure we have everything we want runs the prediction and then responds back to the mms and then to the client and yeah and that's basically what happens on the search maker side so that we are like having um our developer who either uploads a model to s3 or to the hiking phase hub creates our stage maker model provides the endpoint configuration and creates the model this edge maker model here is is our hacking phase hub model and then the end point configuration currently includes the instance tile and the instance count and while i was talking through it the end point got deployed successfully and we can run predictions on it and our example is there's a shortage of capital for project sage maker we need extra financing so i hope mateo can help us on this so we can scale our financing and the model correctly predicts its negative and to showcase the model monitoring a bit i will run over a loop of 500 and uh yeah items and then like send the same request to the endpoint to show you in a minute how model monitoring is going to look like do we have any questions of our name yeah there's a question can the endpoint be accessed outside of sagemaker notebook for example can we hit it with post requests with something like postman yeah so each stagemaker endpoint is exposed as http api but it requires um aws seek for four authorization so the easiest way to access or to work with a sagemaker endpoint is using the awsh sdk where you have different clients for different languages so in python it would be go to free and with the sagemaker invocation client and then you can like invocate it from anywhere you want it's the same for java and go or like node for example and also what you could do is um wrap aws lambda function around your sagemaker endpoint so you would create like a aws lambda function deploy your sagemaker endpoint and then permission the aws lambda function to be being able to invocate your end point and then you could um like request the lambda function from basically anywhere you want and then the lambda function will um call the stagemaker endpoint we have also an example for this i'm happy to share you afterwards sure i'm trying to see if there's any other questions i didn't answer in the chat um oh does inference serving architecture support aws's inf1 instances uh i've read that this is four times faster but hasn't done any tests himself yeah so um shane i'm happy to share with you that we are like heavily working on supporting the instances basically all of them and the release should be like somewhere around end of november so if you wait like a few more months you can use the inferential instances to run sagemaker endpoints and okay our like loop is done and now we can run these the code cell which has like some template strings in it and which will generate automatically for us our um cloudwatch link so if i click on it i will jump directly into cloudwatch um having the metric dashboard for our endpoint open for those who are not aware of or don't know cloudwatch cloudwatch is aws like logging alerting and metric system which or which basically integrates very well with all of the aws services and then you can create like this nice dashboards and set up alarms and in our case we have a dashboard for our um like incoming requests and there we can see we started um when we started the request and like how it's scaled up and the period currently is like how it's like um grouped together is 30 seconds and like the whole timeline is um 15 minutes so you can like adjust it as you want and what's super nice about it so this was created out of the box we didn't need to configure anything at all it will be available for each endpoint you create you will get all of those stage maker metrics and model monitoring metrics and what's super nice um for example if you want to make sure that your model is like not not exceeding a certain limit you could easily set up from here with like this small little notification bell create an alarm and i can directly create an alarm for my end point saying okay model latency my period is like 10 minutes and i had yeah 300 requests and i want to create an alarm when my request number is above like 500 or something and then i can like connect these this alarm to sns and sns can then send like a message to slack teams or even like to your mobile phone as a normal message and that's like super nice because there's like yeah it's not a lot of work to set it up manually and even programmatically it's even same as yeah the same okay going back to our monitoring uh to our notebook okay we have like created our endpoint we run a few requests already have seen okay there's like monitoring dashboard available out of the box but what we really want to do is like to scale up our end point when we get more traffic and how auto scaling basically works behind the scenes um stage maker well i think my screen got a bit small and stage maker starts um instances and these instances are like having a lot balancer in front of it with the endpoint so each request coming in from application service or user comes into the end point gets to the load balancer and then it's going to be split up across the different instances for the example or the endpoint we started we defined one instance and i mean what's nice about the cloud is that you can elastically scale up and down depending on your usage and you are like not wasting resources because you don't need to run like four instances all the time and even if you don't have a lot of load to create our auto scaling group we need to define a resource id we are using boto3 for creating our auto scaling group and our resource id is basically in unique identifier to our end point in this case it's um the end point then our like endpoint name with finbirth bird and our unique identifier and then a timestamp then also we make it applicable for all variants basically and then we can create our autoscaling target which defines okay in which namespace we are we are in sagemaker not an ec2 or ecs we have our resource id we want to scale the desired instance count so we want to scale the number of instances running for our endpoint we have a min capacity and a max capacity so instead of scaling to a maximum of four we could scale like 200 or 2 000 with like millions of requests if we really want to and um so we have like the definition of okay what we want to do but we haven't told aws how we want to do it and therefore we need to create like auto scaling policy which says okay scale my auto scaling group to four instances if something happens and this if is in our case the sagemaker variant invocations per week per instance which is like a nice word for requests coming into my instance and the threshold i'm going to use in this case is 10 so if we are getting more than 10 requests to our instance over a duration of three minutes the auto scanning group will scale up our endpoint to the desired count to like handle all of those requests and then we also defined a scale in and the scale out cooldown which basically makes sure that we are like not over scaling all of the time basically it's a duration or cooldown duration after we scaled up that we are not immediately scaling down if the usage drops like a minute after and then to show this nicely i am running a loop and for 260 seconds just spamming to our endpoint that we will exceed like the threshold for three minutes and then the search maker will start our additional instances do we have more yeah i've got smart i got some more questions for you um the first one is uh which is the normal boot time for sagemaker vm if i want to run gptj or gpt neo model for example in inference i'm worried about boot time how can i make sure i get good throughput uh using batch transform so it's kind of a loaded question there yeah so so normally like creating or starting either batch shop or real-time endpoint takes around three to five minutes really depending on the model size and on the data set and if an instance is warm already by sage maker or not and then after the instance is up and you start your job it really depends on how long the model takes to load for example such uh gptj is like super big i think the float 32 model is around 24 gigabytes or something so it takes a lot of time to load into a cpu because i guess there's no gpu currently who fits the model i think there's an a p4 but it's super expensive so you just like have all of the default three to five minutes for stage makers to start a job and then it depends on what you are doing inside your training or inside your inference batch transform job cool um the other question i saw was uh can these be specified via cloud formation templates what was meant with these i'm not sure who has that [Music] christopher yes so um of course it can be like you can use cloud formation you could use the aws cdk which is like a nice interface for creating cloud formation template you can also use like terraform if you are like using terraform as infrastructures code for your endpoint um it's just that it's currently not supported with the python sagemaker sdk that we went with the photo free ones beautiful and the last thing was do we have any examples of using uh lambda as you were describing earlier yes okay and if those are lying around um we can link to somebody to those later yes cool i think since the script is running i can easily search for it so the lambda example um is not using the sagemaker sdk it's using the aws cdk and then creating like cloud formation templates so if you're not like familiar with cdk which is basically a nice infrastructure as code solution by aws where you could use python and typescript to create your cloudformation templates and then deploy it and there you can find the cdk example for deploying an aws lambda with an api gateway in front of it so you have like an open available http api and also creating a stagemaker endpoint by providing a model id and a task and yet the model of the video will be recorded and to ask answer christian berg's question and you can find it on youtube or you can find it in discord so if you join our discord and then go to the um sagemaker channel when you scroll up a bit there should be the youtube link to the video i'm getting the link right now anyways so great okay i think any more questions folks yeah oh okay still running four minutes a long time you can ask your questions uh on the discord channel as well after the uh even after you have questions even if after you have questions on the workshop or hugging face related you can join join our discord and i'm going to drop a link again uh also philip will announce uh if the if the workshop is uploaded somewhere in the sagemaker enterprise scale nlp channel yeah so we will share again the youtube video i guess like shortly after the whole workshop um today in this course so you can rewatch it again and to answer on a question and you can access i guess you with the server you mean like the um temporary aws account i think it's like 30 minutes until the workshop is done so 45 minutes left i think but you can like easily run those example and creators endpoint with your own aws account and when you go to the github repository in the read media's instruction how you can create the sagemaker notebook instance and then run all of those examples or you can even like run or start the endpoints from your local machine and we have documentation for this as well um great question mikhail you said a target were you took 10 how did you determine a good what a good value is so 10 is definitely not a good value i wouldn't recommend it to you i just went with 10 to make sure or to showcase um to you properly how um how it is how it's going to work i think a good metrics would be um you like create your endpoint to your dedicated um or to this desired like instance type so if you want to go with compute optimized c instances deploy it to there or if you want to go with like gpu deploy just a regular real-time endpoint and then like run a few hundreds of thousand requests to see okay what can um the endpoint handle by itself and what my average like latency i get and there's also metrics for that so if you if we go back quickly to our cloudwatch dashboard and instead of showcasing the sample count we can go to average and we are at model latency there we can see our model latency per request so here we can see the the load test i'm currently running for the four minutes and we can see the average model little c for the cpu is in this case 88 sec milliseconds so we have an instance for c5 we use we think we have four workers so we can send four requests in parallel we have um per request the duration of 88 milliseconds and then you can use like those two value to calculate the throughput of your instance or of your endpoint and then you can set the target value to i think like maybe 90 or something or even 80 so you have like enough time to scale out or go with 60 i think it's like i think you need to experiment it really depends on the model you use for example digital bird is way faster than like a roberta large model and it really depends on cpu or gpu and how many workers you run but then you can like use those numbers to calculate your throughput and then set the target value to what you think is best and going back i can see that it's still running let's do more more questions uh general question what are the best ways to improve inference speed um i think the easiest way is to go from cpu to gpu if you're like just wanting to have quick results and then there are like different techniques on how to optimize your models you can like if you are cpu and also want to have like an easy example you can um just quantize it with using pi torch and exactly as michelle said you can um convert your pytorch or tensorflow models to onyx and then optimize it and then run it there or also inferential which will come our support will come by the end of the week optimizes and my notebook tricked me so it's not like executing anymore when i i can execute the cells and again we can like use the photo free library to describe our endpoint and then print out our the informations and as we can see it has scaled out to for instance and with the desired count for four and if we execute a next line again we are jumping into the stage maker dashboard where we can see our endpoint and then scroll down a bit and we can see okay it has successfully scaled out to four instances due to our request we send and now after not sending the request like in in three to five minutes sagemaker will again scale down automatically and then if you scale up or send more requests again it will scale up again and it's like just these cells two cells we need to execute to enable auto scaling on stage maker for our machine learning models which is like i think personally super easy to use compared to other solutions i think out there okay we have a few more minutes left any open questions i've been swatting these questions like flies in the in the chat so uh i don't leave you very much uh to answer um [Music] are you guys using any specific inf print server for achieving low cpu times on your website examples uh um oh i think they're referring to the inference our inference api versus this yeah so for the inference api like there are two different cpu inference so there's like the cpu acceleration you get when i think being a [Music] lab subscriber and then we have like a default cpu inference and for the accelerated cpus we are using optimize c optimization technique it's not like onyx only it really depends on the model on the task and what we want to do because it's not like you cannot say onyx works for every model every task in nlp like it really depends on the like the model architecture you have is it's in in encoder model decoder model or like a sequence to sequence model with encoder and decoder parts for example they are using on it onyx it's like not very straightforward at all so this question has come up a few times uh philip uh can you clarify uh does sagemaker have https endpoints we can access without lambda yeah so maybe um since we have a few minutes left i could show you how to do it so if we go to our stagemaker console we are at stage maker when you enter stage maker in the search you will enter here and then we can go to inference and end point and on endpoints we can see our endpoint created here it's in service it's real time it's the end point where we run our inference and at the top it includes an http api so in url so that's the https and point for our stage maker uh model but you cannot just like open any res client and send a post request to it because these endpoints are secured by aws seek for authorization and we can like easily click learn more about it where it is described how i described how you can and like invoke those endpoints one example is what we are doing with the python stagemaker sdk is using the predictor.predict which is creating those signature behind the scenes and using our input but there are also like different options on how to do it um so for example if i scroll down here not this edge maker sdk is used here we are using the photo free library where we create a client for our stagemaker runtime and then we can run the client.invocate method this like structure for creating those authorizations4 headers is available for each aws sdk so it's available for java for node.js for go for python for i think c plus plus or something or maybe.net as well so you can definitely i would recommend to use those sdks if you directly want to interact with sagemaker endpoints and not using the sagemaker sdk and of course you could also like create your own header without the sagemaker sdk but creating those thick for authorization headers not very like straightforward so go with the sagemaker the aws sdk and then use the um invocation endpoint methods or just stick with the python sagemaker sdk and to answer yourself a question what is both free bottle free is basically the aws cdkf sorry the aws sdk for python so with photo free you can interact with all aws services programmatically um how about this question can we serialize or pickle a deployed predictor and use it in another environment i'm not really sure what team means well i think you can you can just take the the assets the model assets you're already kind of uploading model assets so you could just use those anywhere you want in any environment right yeah so we are like not doing any specific we are loading a model from dehydrating phase hub and deploying it as endpoints so instead of deploying those model to stage maker you could deploy it like to your local machine or to like something else [Music] and um all of those high interface models can be uh can be deployed this way not all of them we are working on it so the hiking phase hub is um as you probably have seen it's not like limited transformers only we have like a lot of new uh models from different frameworks like um fast ai psychic learn um flair and speech brain for example but all transformers compatible models which support nlt tasks are currently deployable so all of those i think 12 or 13 different nlp pipelines are deployable and we are working on enabling um speech pipelines like automatic speech recognition then you could use wave to work or uber as well and are also working on image classification pipelines where you could then use vision transformers and those should be also available like by the end of the year so we are trying to expand the sagemaker experience to the up those other modalities as well uh philip there's a question about uh how to deploy real-time streaming uh model with aws they're um they're asking specifically about wave to vect ii but i think it's more about the architecture um can you check out that question there yeah um that's that's a great question and like a super great use case to be honest so you said um or for those who didn't read the chat it's like um they try to deploy a real-time streaming wave to wack model with aws and the initial idea was searchmaker and then websockets with lambda but it didn't work out well so um web circuits with london is a great way to create like real-time applications um i think you would require a lot of work inside the lambda to like send those requests and i think um the issue currently with wave to wek or with the asr pipeline is that it's like not yet suited for like streaming so the best way currently to use the automatic speech recognition pipeline inside transformers with wave to work is have an audio file provided and then they get the transcription back since streaming is like a bit different kind of work since you would need to chunk it properly and then since you are not like sure okay which chunks you send um it can create some strange results but i'm happy to share that the open source team is like working on it on enabling streaming and if streaming is enabled inside the open source and for the speech recognition pipeline it would also be available for sagemaker and then i think we will find a way on how you can properly stream inputs from ite outside of stage maker to sagemaker either through web sockets with lambda or like natively using the sagemaker sdk and i saw another question from yop can you deploy private hybrid phase models yeah you can and so to deploy private hugging phase models we had the hub configuration with our model id and with our task and to deploy a private model um you just need to add a third parameter which is hf token i think but you can look it up in the documentation and then you would need to provide your hiking phase token which has permissions to access the private model and then those tokens will be passed into sage maker and search maker will then use the token to load your private model so it's just like providing um your permission to sage maker for maker to be able to download the model beautiful uh there's another question on using weights and biases um uh in the last session we actually did the training we didn't do that here but when you're doing training there's the training arguments object and there you can just add wand b to the report to keyword argument which can be found in our documentation for that training arguments object and i believe 1b lets you or is expecting you to have a certain end type file um it's you can read about it in their documentation um where they specify how you can give your api key um on sagemaker but uh there's there's a whole write up on it um by the one b team too so you can check that out there okay i guess we have like we are almost to the hour but i think we can answer one more last question so who wants to have the honor okay there we got one so can you deploy custom models even they are not uploaded to the hub right yeah so you can like deploy any transformers nlp compatible model to sagemaker either through the hub directly or through s3 and in the documentation you can find instructions on how to do it there are like notebooks on hanging face notebooks repository where you can also find examples on how to deploy a model from s3 to sagemaker great then thank you for participating and yeah happy to talk to you um hopefully seeing some of you on discord or on the hugging face forum and then until next time thank you again bye thanks everybody