Configuring and Managing Amazon S3 Replication - AWS Online Tech Talks

Key Takeaways

hello and welcome to our AWS online Tech Talk on s3 replication my name is mo Farhad and I'm joined by my colleague groovy dang on the s3 product management team today we'll learn about the use cases and benefits of s3 replication review the latest additions to s3 replication capabilities and learn how to configure replication policies to help you meet your compliance security and operational efficiency requirements let's get started with a review of our agenda first we'll review why you should replicate your data on s3 discuss exciting new capabilities of s3 replication and then we'll review how we can configure them for your needs and share a quick demonstration finally we'll be happy to answer any questions that you may have let's get started Amazon s3 customers receive a very high level of availability and durability for their s3 data in every AWS region customers often have compliance or business requirements to replicate their data to second region hundreds or even thousands of miles away from their primary location s3 replication provides an automatic mechanism to make identical copies of your objects in a destination of your choice replication can help you meet data resiliency compliance requirements or place copies of your data close to internal users for operational efficiency or close to external users to minimise their access latency those use cases involve replication across AWS regions other needs often involve replication within the same AWS region for example s3 replication can be used to automatically copy data between your production and test buckets or accounts helping your teams be more agile or to simplify analytics jobs by aggregating logs from multiple services in different buckets to a single bucket for analysis finally replication can also help you protect against unintentional deletes by keeping a copy of your data in a different account while meeting data residency regulations that require that data to stay within a specific AWS region this can all be done while replicating data to a less expensive infrequent access storage class saving you costs along the way let's review how s through your application helps you meet those needs in more detail next s3 replication is an automatic and asynchronous process that makes an identical copy of your object that means it retains all metadata that is on the source object including important fields such as the object creation date and time and eversion IDs now s3 replication has always worked across AWS regions and customers have asked us to extend that capability so that they can replicate in the same AWS region we've recently announced exactly that and now you have the choice of a destination bucket in the same region as the source now let's take a closer look at the capabilities of s3 replication s3 replication is very easy to setup as will show you in a short demo later in our talk s3 replication gives you the choice of replicating entire buckets s or a set of objects under a given prefix or even specific objects that you identify with tags of your choice replication begins as soon as you upload an object to s3 that is covered by a replication policy making an identical copy of that object to a destination bucket once you've selected that source dataset you select that destination bucket which as we've mentioned earlier can now include buckets in the same region as a source to enhance security you can choose to change the ownership of your replica copies and also to replicate them to another account so that no single user or single account can erroneously or unintentionally delete all copies of your data depending on your use case for the replica copy you can choose to keep it in a frequent access storage class or replicate it directly into an infrequent access or an archive storage class such as s3 glacier you can also specify different lifecycle policies on your source and replica objects to optimize costs based on your access patterns in both the primary and secondary locations if those access patterns differ now since launched in 2015 s3 customers have replicated trillions of objects and exabytes of data and today we're making this robust capability even more powerful with s3 replication time control we're pleased to announce that Amazon s3 replication time control is now available in all AWS commercial regions customers are often interested in the time objects take to replicate to their destination this time can be influenced by factors such as object size count and other attributes when you need additional control and visibility over replication time you can now use all of the capabilities of s3 replication with a new 15 minute replication time service level agreement and a set of cloud watch metrics and event notifications that are part of s3 replication time control let's find out more replicas time control gives you predictable replication times such that most objects replicate in seconds 99% of objects replicate within five minutes and 99.99 of objects replicate within 15 minutes this replication performance is backed by an SLA that covers the replication of 99.9 percent of objects in each replicating region pair in any billing month in the unlikely case that the SLA is not met you'll receive a CREP on the replication time control charges data transfer and request charges and the destination stores charges associated with the objects that did not replicate within 15 minutes and the billing period affected you can view the full service level agreement on the AWS SLA website s3 replication time control goes further and gives you a set of metrics that you can use to monitor replication for each replication rule these metrics tell you how many objects and bytes are pending replication and the maximum replication latency of the objects covered by your replication time control policy you can view these metrics in the s3 management console or in cloud watch and set up alarms to notify you when any of these metrics exceeded threshold of your choice these cloud watch metrics are aggregated for each replication rule now you may also want to monitor replication at the object level which you can with the s3 replication events that we'll review next in addition to the cloud watch metrics that we've discussed s3 replication time control will publish events in the rare instance that an object doesn't complete replication within 15 minutes replication time control will also follow up with another event that tells you when that object did successfully replicate this destination and like other s3 events that you may have used in the past you can receive these replication events using the AWS simple queue service SQS the simple notification service SMS or use them to invoke a lambda function as needed I hope you're as excited as we are about these new capabilities of s3 replication now let's listen to my colleague Ruhi as she shows us just how easy s 3 replication and replication time control are to setup Thank You mo well now that you've learned about what s3 application time control does and all the benefits it gives you we'll dive right into how to set it up just like you would while setting up s3 replication first select what the source and destination region should be based on your compliance and business requirements next you want to decide what dataset to replicate whether you want to replicate an entire bucket or prefix a tag or a combination of these now once you've chosen your dataset you can enable replication time control you can setup either a new replication rule with replication time control or add it simply to an existing rule once you enable this feature you will begin to see replication metrics for each of your rules so let's start with what a basic replication configuration looks like when you set this up and then we look at what it looked like when you enable replication time control on your policy and then finally we'll follow up with the demo on the s3 management console so this is just a basic replication configuration it sets up a new rule and tells s3 that the object in the tag prefix will be replicated to the destination bucket that you specify moving on to enable replication time control all you do is add another section that's highlighted here to your replication policy now this is what you do when you were setting this up programmatically there are many more policy examples that you can find in the s3 replication documentation including how to set up cross account or cross storage class replication you can also perform this setup in the s3 console which we're about to see a demo off so before I jump into the demo straightaway um what we'll do here is we will create two buckets a source pocket and then a destination bucket in a separate region and then we will set up replication on it we'll upload a few objects and then we will see how the metrics look like we look at replication metrics and then we will even set up an alarm on one of them so let's let's get into this so let's create a bucket let's call it 11:25 test source for today's date let's set this up in Northern California we move on to the next stream we turn on versioning which is required to set up replication on the next screen block public access is enabled by a default we keep this setting let's move on verify your information and create a bucket so we have the source now let's create a destination bucket we'll name it 11:25 test test and let's pick a region that's a little far away say Mumbai moving on to the next screen again we enable versioning on this pocket and we'll have block public access that's set up by default and then you move onto the next screen verify your information and click on create bucket so because we now have two buckets a source and a destination bucket let's go into the source bucket into the management tab and click on the application this will help us set our application configuration we want to set it up on the entire bucket go to the next screen and select your destination bucket that's the one we created and then let's check replication time control which is the feature we discussed on to the next screen we create an IM role which has the permissions required to replicate your data let's call it my role 1 and verify the information and save the role so here you can see all the details of the replication rule my rule 1 that I just created you can see the application time control is enabled so let's go to the bucket and upload some objects I have a bunch of test files here that I'm gonna drop into this bucket you can see these are 12 files a total of 2 point 8 gigabytes of data um I'll fast forward a bit as these objects upload um so we have them here you see those twelve objects that I uploaded let's see what the metrics look like we go to the management tab under metrics and click on the application let's look at what happened in the last 1 hour and then select the rule ID that we just created you display the Crofts and there you have it so the first one is the application latency it's telling you what was the maximum time that it took any object to get to the destination and you can see the max here it is about 40 seconds and you can see those spikes as different objects got uploaded and replicated to the destination and it's going down to 0 at the end since everything reached the destination next we have bytes pending replication so you can see the peak here is about one gigabyte so objects were uploaded and replicated as we went along and this is giving you a minute by minute detail of that the final one is um operation spending replication these are the number of operations that have not yet replicated and they're also going down to 0 as everything reached the destination let's set up an alarm so let's pick this operation spending replication metric and set up an alarm in cloud watch so you click on that quick link it'll take you to the cloud watch console where you can set up an alarm so you see the metric that I selected written here you can see the metric chart that we saw in the previous screen right here we have the source and destination bucket we want a one-minute frequency it is the rule ID that i created my roll 1 and then we want to set up a threshold I'm gonna pick an arbitrary number of hundred here which means that every time the number of objects that were pending replication exceed hundred I want an alarm the other thing to note here is the missing data treatment should be set as treat missing data as ignore while you're setting this alarm we want to create a new topic let's give it a name let's call it say alarms topic 1125 for today's date and we'll enter a test email ID here for the notification we call a test topic at amazon.com and we'll set up the topic so the topic is setup you've indicated what threshold is moving on to the final screens of setting up the alarm you give it a name let's call it alarm 11:25 operation spending to specify what I was setting up the alarm on and on the next screen here you can see the red line which indicates the level of the alarm which was 100 and my blue small information here which was the actual values of the metrics you see all the information you set up here it's all there verify the information and then simply create alarm now the alarm has been set up you can continue to monitor this update it and so forth um so that's it folks for the demo now that we've seen this demo and before we get into Q&A mo why don't you tell me why would a customer want to use replication time control well Rudy many customers are under regulatory requirements that ensure that they have a minimum level of business continuity and data protection planning that often includes maintaining a backup copy of their data that's sufficiently resilient and geographically distant from their original copy of that data so that any event such as a natural disaster or a bad actor cannot impact the availability of this critical data customers also may make commitments to their own end users that make their data available in an resilient against such disruptions and with replication time control customers have the confidence of that predictable replication time that they could use to meet such compliance and business requirements now replication is one thing there's ability into a replication is another particularly in these often stressful moments when a failover is necessary maybe Ruby you could walk us through how a customer could use replication time control metrics in such a scenario sure as to consider a failover situation where a customer had set up replication time control on their data and they're no longer for whatever reason no longer able to access the primary region what you could do as a customer is you'd look at the replication latency metric which gives you the maximum latency at any given point of time it's a four minute level metric so you could look at the time stamp and see what their application latency was at that time stamp and subtract it from the time stamp to get the point in time that you want to failover to so let's say at 9:00 a.m. your replication latency was 10 minutes which means that you could take 850 as a good point in time to reverse your applications - now there was not a metric we talked about - other metrics we talked about byte spending replication and operation spending replication at that same time point you could look at these two other metrics and figure out what was the number of objects and the total size of objects that were still pending the application together all of this information will give you a higher confidence in what that failover point should be that's great really these are unique capabilities we're tremendously excited about while the questions are coming in let's pull all this together in a customer example you may have a requirement to store a second copy of important data securely before your application or employees act on it you can use replication time control to automate this process and replication events to report on compliance you begin by configuring replication time control to replicate your objects to a second account with ownership override so that no single account can unintentionally delete all copies of your data and to lower costs you can store the replicas copy in an archival storage class such as s3 glacier you can then configure s3 events and lambda to store records in an RDS table that tracks when an object was stored in the source bucket whether its replication time exceeded 15 minutes and if so when that object successfully replicated this information allows you to regularly report a compliance to internal or external auditors Ruby will walk us through the setup now we've seen what setting up replication on the console looked like but let me walk you through how you do this programmatically now as more described in the use case we want to create a second copy of your data with a different ownership control for this you'd start by first setting up buckets a source and a destination bucket that are owned by different AWS accounts you can use the create bucket API or a number of other ways to do this programmatically by logging in from different EWS accounts once you've created the two buckets the next step is to add permissions to your destination bucket to allow changing the replica ownership in replication the owner of the source object also owns the replica by default and so to change this behavior the destination bucket owner needs to provide permissions to the source account owner to change ownership here's the bucket policy that you would attach to your destination bucket the owner of the destination bucket must grant the owner of the source bucket permissions to change replica ownership in a bucket policy the principle element that you see here specifies the user account service or another entity that is allowed or denied access to a resource reading through this policy you can see the principal here is the source account because we want to allow the source account which is the default owner of the objects to change the ownership you can see the effect is allowed and the resource we want to act on is the destination bucket which is specified here towards the end of the policy and finally the permission that allows the source bucket owner to change ownership is highlighted here its s3 object owner override the bucket owner action now that we've created the two buckets and attached the destination bucket policy the next step is to create an iamb role Amazon s3 cannot replicate objects without your permission you grant permission with the I am rule that you specify in your replication configuration s3 then assumes this role to replicate objects on your behalf by default all your amazon s3 resources including buckets objects and related sub resources are private only the resource owner can access the resource to read objects from the source pocket and replicate them to the destination bucket s3 needs permissions to perform these tasks you grant these permissions by creating an iron role and specifying that role in your application configuration this policy right here will grant the permissions specified here to the user or actor that assumes the role it contains permissions you are allowing or denying and the resource on which the permissions apply starting with the first block up here you can see the effect is to allow and the resource of the source bucket to replicate objects s3 requires explicit permissions to read objects from the source bucket and also to read their application configuration attached to the source bucket so that s3 knows which objects to replicate on your behalf and the permissions you see in the first block do exactly that moving on to the second block you see the effect is to allow and the resource is the destination body s3 replicate object and s3 replica delete a permissions for the respective actions on objects in your destination bucket these actions allow s3 to replicate objects or delete markers to your destination bucket finally since we want to change replica owner you grant Amazon s3 permissions to change replica ownership by adding the highlighted permission here object owner override the bucket owner now that we have all the permissions in place we could set up the application configuration which brings us to the last step where you add a replication configuration to your source pocket we start by specifying an iamb role that we created previously you can see that in the first line of the configuration here this gives s3 the required permissions to replicate your data you can have multiple rules in your application configuration if you want to select a different subset of objects in each rule you will specify a filter that selects a subset of objects here you can see we have one rule and the filter is prefix RTC data the status of the rule indicates whether it's enabled or disabled if a rule is disabled Amazon s3 doesn't perform the actions specified in that rule the priority right here indicates which rule has a priority when multiple rules apply to the same object coming to the destination block this is where all the fun happens you specify the name of the destination bucket next what we're doing here is we're changing the storage class of the replicas to glacier this helps you reduce costs while durably storing a second copy of your data in a colder tier storage class the next block you see here that's also highlighted access control translation changes the owner of the replicated objects to the destination account the final two sections in this configuration enable the feature replication time control that we talked about in detail today in this particular rule it also sets a 15-minute threshold on the replication events the 15-minute threshold is the only acceptable value today the setup is completed after this step and any objects that you upload to your source pocket under the prefix RTC data will now be replicated to your destination bucket the estimation account will inherit ownership of those objects and the objects will be stored in the glacier storage class great now that we've configured the replication policy let's take a look at how we can use events to keep a record of any objects that deviate from the replication time control SLA s3 replication time control will publish an event for any object that doesn't replicate in 15 minutes like other s3 events you can invoke a lambda function based on these replication events and use lambda to create a record of these events in an RDS database table that you specify for example you can use the s3 event object created put to create a database record of objects stored to your source button and the replication event operation missed threshold to create a record of the name size and version ID of any object that took longer than 15 minutes to replicate you can see a portion of the event format here you may want to also record the replication rule ID destination bucket and s3 operation highlighted here you can also append that object record with information from the follow-on event operation replicated after threshold that tells you when the replication tasks completed for that object let's take a look at the event format here notice the added replication time element highlighted that element will tell you the time in ISO format when replication completed you can then use these records to build a regular compliance report that documents any deviation from the 15-minute replication threshold and provide that to your orders with that we'll be happy to take your questions