How to use Amazon S3 Batch Operations

Key Takeaways

hi my name is Rohan Thomas and I'm a senior product manager with Amazon S3 today I'm looking forward to talking to you about how to use Amazon S3 batch operations in this video I'll provide a provide a guided tour of Amazon S3 batch operations a feature designed to perform large-scale operations across tens to billions of objects in Amazon S3 we'll see how it works and show you how to set up permissions create a job and how to manage and track a job all in the AWS Management console with that let's Dive In first let's explain what Amazon S3 batch operations does as your data footprint expands you'll find it increasingly valuable to make changes across many objects at once especially if you store exabytes of data or billions of objects that's why customers rely on Amazon S3 batch operations a feature which makes it simple for customers to manage tens to billions of S3 objects with just a few clicks in the S3 Management console or a single API request with S3 batch operations you can ask scale modify object metadata and object properties such as replicate or copy objects between S3 buckets replace object Tag sets modify access controls to sensitive data restore archive objects from Glacier flexible retrieval and Glacier deep archive as well as invoking AWS Lambda functions across all of your S3 objects the great thing about S3 batch operations is it eliminates the need to spend months building and maintaining an application that calls apis across a set of Target objects to get started all you have to do is create a list of objects you want to modify or take action against you can do this easily by using an S3 inventory report or creating your own list of Target objects then select your desired operation from our pre-populated menu in the S3 Management console once you've created your job you can use the console or the S3 apis to view progress and prioritize your jobs when your job is complete you'll review a notification and a completion report that lists all the changes made to your S3 objects let's start by looking at setting up permissions for SP batch operations in this segment we'll be looking at the steps required to create an AWS identity and access management role and this defines the set of permissions you pass to Amazon S3 batch operations in order to perform a requested job for this demo we'll be operating in the AWS IAM console to get started we'll select roles and then we'll click on the batch operations role that we're dealing with I'm going to show you how to modify an existing role but the same steps could be used to create a new role first let's look at the permission policy that we have already let me open it up we can see that it's already populated with the bucket that we're using but all you have to do is actually copy and paste from the Amazon S3 batch operations documentation we can refer to the documentation page for granting permissions for M for Amazon S3 batch operations you can just copy and paste to Json specified for replacing object tags and then all you have to do once you use that Json is you would have to replace the Manifest location as well as the completion report location in our case all those buckets are the same but you may want to choose and specify different buckets for your manifest and related completion reports just inspecting the Json the first section gives access to tagging the objects the next gives us the ability to read the Manifest and lastly this gives permission to write the completion reports after doing this we can save the changes and move back to the role creation the next step is the trust relationship this is what allows Amazon S3 batch operations to assume the role and perform the action specified once again to create this we have to refer to the documentation you can just straight copy and paste to trust policy and save it for your AWS IM role now let's look at creating a job in this segment we'll create an Amazon S3 batch operations job and this job will tag objects in an Amazon S3 bucket we'll start by selecting a manifest which is the list of objects and then we'll specify the parameters for this particular job so let's get started first we'll start in the Amazon S3 Management console to select an Amazon S3 inventory report as to manifest for our job and Amazon S3 inventory report lists all objects in a given bucket or prefix making it a very good fit for creating a job that acts on your existing storage an inventory list contains a list of objects in the source bucket and metadata for each object objects are sorted and ascended ascending order based on the key names we start by selecting our bucket and then we select a management Tab and finally we scroll down to the Amazon S3 inventory feature there we see my existing configuration and can select create job for manifest and this starts creating our job that takes us to the job creation workflow and Amazon SUV batch operations with the latest Amazon S3 inventory report selected as our manifest here we verified the region for our job and this will be the destination region for copy operations and the region where our objects are stored for all other types of operations we then verified that the Manifest we selected for the job is correct of note we have a few options for the Manifest the first option is using an S3 inventory report like we are the second is providing a manifest in CSV format and lastly in the special case of S3 batch replication we can have S3 batch operations generate a manifest for us using the under underlying S3 replication configuration now we scroll to the bottom and we click next here we can see the type of operation that we want to perform we're going to select replace all object tags and now we're going to actually put the key value pair for our tags I'm going to put a value of project and sorry key of project and the value of archive and this is what we're going to be using to later go on and create a life cycle policy to later archive our objects within our bucket after doing this we're going to click next at this step we can give our job a description to help identify and search for it among other jobs since we're doing project archive I'm going to add that to the description of our project next we can prioritize this job above or below others this capability is specifically designed to let you control what jobs are active at any time and which ones to pause if a higher priority job needs to run please note that a higher value here indicates a higher priority for now we'll just leave this job at priority 10. in this next step we will specify that we want a completion report is actually already selected generate completion report but I'm actually going to just select that I want fail tasks only and this this option helps me to identify and correct any issues that prevented tagging my objects I'll then go and specify where I want this stored so I'll browse my location and select the bucket where I want the completion report stored now we need to specify the AWS IEM role for Amazon S3 batch operations the one it should assume to perform this operation this includes the permission to read the Manifest tag the objects and write the completion report to my bucket if you haven't yet created an AWS IAM role for this job there's guidance available when you click the drop down which is titled view IEM role policy template and IAM trust policy earlier in this video I walked through the steps on setting up permissions for Amazon S3 batch operations so since I already have a role I'm not going to create one right now I'm actually going to go and search for the one I've already created and use that lastly I have the option to tag the the job for organization purposes I'm now going to proceed without a job tag and move on to the next step by clicking next here I verify all the parameters are as intended and then finally I scroll to the bottom and click create job now as our final segment let's move on to managing and tracking a job in this segment we'll be exploring the actions you can perform on your jobs once they are created these include changing a job's priority confirming a job cloning a job canceling a job and lastly tracking a job so let's get started first let's look in the Amazon S3 Management console we'll be going to the Amazon S3 batch operations tab to see a generated list of jobs here we see a list of our jobs 100 at a time which can be filtered by their job state region and other parameters let's start by looking at all of our active jobs so we do that by selecting the the drop down that is currently labeled as all status types and we're going to click in progress and here we can see all of the jobs that are running and the progress that is made and we can see that all of these job priorities are at 10. next we're going to go go look at the jobs that have not yet started and we're going to do that by chaining the in progress drop down to not start it so we can see that there are some jobs that are in the ready State as well as some that are preparing and some that are waiting for confirmation the jobs that are in the ready state are waiting for an active job to finish running before they can activate if we want to prioritize some jobs or over others I can change the order so the important ones act sooner with all the active jobs at priority 10 all change one of the jobs to priority 12. so if I do that by selecting the job and I'm going to click actions and I'm later going to click update update priority so here I have the current priority at 10. I'm going to update it to 12. and click update priority we can see that that job has not moved away and now it is getting into the ready state among the jobs that are not yet running you can see that some are in the waiting confirmation these jobs are awaiting user action before they can run please note that all jobs created through the Amazon S3 Management console automatically moved to this state however for jobs created through the AWS sdks or CLI this job state is configurable as well as when you use S3 batch replication you have the option to run your job automatically without waiting for confirmation to confirm a job what we have to do is we have to go and select a job and then click run job we scroll to the bottom and click run job once again and the job is confirmed to copy the parameters from one job to another you can clone a job in the S3 Management console this action pre-populates all the job creation Fields with the details from the other job you this can be extremely useful when the jobs are very similar and only the Manifest or the operation needs to be modified or to retry a job after the error was corrected so to do that you would click select an existing job and click clone job next you can cancel a job if it is no longer required or if you need to change any of the parameters of the job if the Manifest operation details or other parameters of a job need to be altered you need to create a new job so the only exception to this is changing a jobs priority if a partially completed job is canceled any work that has been completed remains and does not roll back so to cancel you would just click actions and then click cancel job with S3 batch operations you can view and update job status add notifications and logging track job failures and generate completion reports in the console we can see current job statuses as well as percentages for the percent completed as well as totals failed rates when using the SDK and CLI you can use the describe job API to retrieve the configuration parameters and status for a batch operations job while a job is active to describe job API monitors is progress similar to the S3 Management console in addition you can also capture review and audit batch operations activity using AWS cloudtrail because batch operations use existing Amazon S3 apis to perform tasks those tasks also emit the same events that they would if you call them directly thus you can track and record the progress of your job and all of its tasks using the same notification logging auditing auditing tools and processes that you already use with Amazon S3 thanks for watching be sure to check out Amazon S3 batch operations feature page for more information on S3 batch operations