Understanding High Availability and Disaster Recovery Features for Amazon RDS for Oracle

howdy my name is Michael Baris and I'm a senior database engineer with the Amazon relational database service thanks for joining me today in this talk we're going to take a deep look at high availability and disaster recovery with Amazon RDS for Oracle like most Oracle DBAs I've spent many hours configuring servers and storage and struggling to get complex availability architectures working so much so that the database itself can seem like an afterthought RDS automates that tedious infrastructure work so I can spend my time on what's going on inside the database instead of working to keep the infrastructure running but it's still important to understand how things are put together under the hood in order to decide which features to use the primary audience for this presentation is Oracle DBA S and architects who want to understand how best to use RDS so I'm going to assume you are familiar with Oracle databases this is a deep dive so I'm also going to assume you know a little bit about AWS is region and availability zone architecture the Amazon relational database service and have a high level understanding of features and concepts for high availability but let's take a quick glance at each just two reefa miliar Eyes ourselves the region and availability zone architecture is foundational to how we think about availability at AWS today this infrastructure spans 22 regions and 69 availability zones globally with more already underway this design comes from Amazon's experience as a fast-growing online retail company we realized that individual data centers will inevitably experience failures and that the best approach to availability is to rely on a cluster of data centers with data synchronously replicated between them to expect failure and to recover from failure as quickly as possible with automation we also learned that the availability factors for data centers very far apart from each other are difficult to account for so regions are built as isolated clusters with the services in each region running their own dedicated control planes I like to think of a region as a geographic location where you can build highly available applications to achieve this each region is built from a cluster of data centers organized into availability zones or AZ's availability zones in a physically separated from each other by a meaningful distance and connected by high-bandwidth low-latency fiber this means that AVS are close enough that you can synchronously replicate between them but far enough that failures affecting one are not likely to affect another which region you use is an important decision typically you want to choose the region that is closest to your customers to ensure lowest latency or choose the region based on geographic location when data locality and sovereignty are important which availability zone you use is not important Easy's within a region should generally be treated as interchangeable and your application servers should spread across as many as possible to ensure the highest overall availability hopefully you're familiar with the Amazon relational database service or RDS RDS is a managed database service built on top of other foundational AWS services like Amazon Elastic Compute cloud Amazon elastic block store and Amazon simple storage service or s3 like these other services you use the AWS console or command line tools to create a resource that is automatically provisioned for you in minutes with RDS this resource is a database server running the database engine and version and configuration of your choice choose from open source engines like Postgres my sequel and marija DB and commercially licensed database engines like Microsoft sequel server and oracle database automation handles the health of the database software and the infrastructure runs on this lets DBAs focus on the inside of the database you manage the configuration performance users and data just like you would on premises RDS instances are easy to administer automation handles the heavy lifting for the infrastructure with just a few clicks you can restore backup scale compute resources up or down upgrade to the latest security patches pause your database for a few days or configure monitoring and alarms based on server Health metrics RDS has built-in features for availability and durability which is what we're going to focus on today and it supports the latest ec2 instance classes and EBS volume types letting you scale up the size and power of your database as needed finally let's talk about some relevant terms recovery point objective or RPO refers to the point in time to which data can be recovered RPO is a design goal or a business objective and helps you to understand what a system is capable of and what your requirements are in terms of data loss during a recovery event recovery time objective or RTO refers to the time it takes to perform a recovery action understanding features in terms of RPO and RTO helps you make decisions about what's possible and what features to use the green line here represents the ongoing activity of my database and you can see that some sort of incident has disrupted it if I have to restore from backup then the recovery point is in the past and my data loss is determined by how far back I had to go recovery point objective represented here in blue describes the maximum amount of data loss expected with a particular recovery action the recovery time objective represented in orange is the amount of time the recovery action takes to get me back to production back to the Green Line when planning and designing your database architecture you start by identifying the RPO and RTO that your application requires and then use the feature or recovery actions that satisfy those requirements as we look at the high availability and disaster recovery features of RDS we'll talk about how RPO and RTO relates to each because one copy of your data would be a single point of failure methods must be used to make additional copies of your data by replicating it elsewhere in some cases the replication may be synchronous meaning it has to be safely written to more than one place before it can be considered committed or durable that represents a trade-off of performance for durability which typically gives synchronous replication in RPO of zero with asynchronous replication applications don't have to wait for replication to complete sir performance is not impacted the trade-off here is increased vulnerability to failures that occur after data is made durable locally but before it can be replicated remotely which translates to nonzero RPO replication can also use either physical or logical data structures with physical replication binary data typically raw data blocks or binary change vectors is directly replicated to the other side this is usually faster but can be less flexible because the data has to be applied as is with logical replication binary data is converted into logical records and replicated as insert update and delete sequel statements this takes additional time and processing but allows you to control what gets replicated and how it looks on the replicas now that we have a firm foundation on AWS RDS and recovery terminology we're ready to talk about how Amazon RDS takes care of replicating your data within and across AWS regions and how that affects the RPO and RTO of a few different scenarios high availability refers to system designs that maintain the availability of a resource in the face of underlying failures in this section we'll talk about how RDS keeps customer workloads up and running there are two important areas to consider when looking at high availability for your applications first what are your high availability requirements and second what else impacts the availability of your application these seem obvious but are really important for getting things right we tend to just think that everything should be up all the time but high availability doesn't come for free it's an intentional trade-off of cost and complexity for uptime I find that customers are most successful when they're able to clearly define the RPO and RTO requirements for each workload when analyzing existing applications it's important to think that of things like what's your current level of availability how are you measuring that and are you meeting your current goals what is the required level of availability what is the business driver for that requirement and is it reasonable and once you settle on a design go back and review whether it meets your requirements if it doesn't why is that and is that reasonable not all applications require high availability and there are practical limits to the availability of any one database but starting from business requirements will help you make the right decision for your particular use case now the database is just one part of your application so it's also important to understand the other things that impact availability for example are your application servers spread across all of the availability zones in a region or do you have hard dependencies on some service or resource that is only configured in a single AZ or a single region or for the portion of your app that talks to the database how does that handle disconnects when the database fails over well you should be able to answer these questions it's also important to actively test your applications to understand how they are impacted by different failures RDS provides high availability with what we call the multi availability zone option multi AZ configurations synchronously replicate your data to separate infrastructure in a second a Z the configuration is continuously monitored and failover happens automatically typically taking a minute or two this allows us to offer at 99.95% uptime SLA and because we're replicating at the storage layer this works with both standard edition as well as Enterprise Edition in an RDS single AZ configuration the compute note and storage volumes for an instance are created in the single availability zone if the compute node fails or the database crashes RDS automation takes care of getting things back up and running usually within a few minutes but sometimes longer if an availability zone itself experiences failure though something that's rare but can happen and it's the AZ where your instance is running your application may be unavailable until the problem is solved and your instance can be brought back up for workloads that require a higher degree of availability RDS multi a ze creates a secondary instance with the exact same ec2 instance types an EBS volume configuration in another availability zone it then sets up replication maintaining an exact duplicate of your data on the secondary RDS multi AZ uses synchronous physical block level storage replication to ensure that the secondary always has an exact replica of the primary data when Oracle writes to disk it waits for the storage system to acknowledge the right before it responds back to the session that its changes have been made durable with multi ez the same thing happens except the storage layer transparently replicates the change to the secondary instance and only acknowledges the right after it's been sent to storage and acknowledged on both sides because ACS are relatively close together we can do this right synchronously to reduce the risk of data loss it can increase right latency by a small amount right latency is now the cumulative latency of both the local and remote writes so it's something you should test carefully with your application but most customers find this to be a reasonable trade-off note that the secondary does not actually host a running database instance so you can't connect to it for read scaling but in the case of Oracle it does need to be licensed if the secondary instance fails or becomes unavailable will automatically detect that fix the problem and restore high availability all with no impact on the primary instance or application sessions if the primary becomes unavailable RDS automatically converts the secondary instance into a primary allowing your applications to reconnect RDS takes care of reinstating the failed primary as a secondary instance and restores high availability notice here that my application servers are running in all of the available AZ s3 in this case even though my database is only running on to a Z's remember the database is just one part of the application and there are other edge cases to plan for by using as many a Z's as possible your application is best able to absorb the impact of losing it an entire AZ also note that the primary and secondary RDS instances have the exact same configuration in each availability zone this mean that means that there's no need to fail back after the fault has been resolved so there's no need for a second outage with multi AZ your instance will run in AZ a for a while then an AZ B for a while and so on again which a Z you use is not important now let's look at what happens during that failover after the primary becomes unavailable it takes our automation a few seconds to detect and confirm the failure to ensure we don't failover unnecessarily the failover itself consists of two key operations in parallel database crash recovery and DNS propagation in the crash recovery phase the database starts up and returns data files to a consistent State the process is no different than if the database had simply crashed in fact the database doesn't even know it's running in a different place the very last transactions committed in the moment immediately before the failover were synchronously replicated so no data is lost in a failover scenario any uncommitted transactions that were in flight when the failover occurred are rolled back by crash recovery and no different than if the database had just crashed the failover also involves a DNS change to point your instance end point to a new IP address this way applications don't need to be reconfigured or modified at all after a failover they do need to try to reconnect though and they do need to do a DNS lookup when doing so applications that use connection pools work really well here since the pool automatically retries connections until they succeed allowing them to resume work as soon as the listener is up as soon as the database is open and the listeners up clients that resolved a new IP address for the end point are able to connect in most cases that process takes one to two minutes so by maintaining a second copy of your data on physically separate infrastructure and automatically failing over RDS multi a-z can offer a 99.95% SLA going back to our high availability terms RDS multi a-z typically operates with an RPO of zero and an RTO of one to two minutes you we've seen how RDS can keep your database up and running now let's talk about how it helps you recover from larger scale failures let's say a disaster is anything that affects the availability of your database beyond what multi a-z can handle for example a region level failure or multiple simultaneous hardware failures or perhaps an application bug that affects the integrity of your data as with high availability there are some things for you to think about when planning for disaster recovery we'll talk about how RDS enables disaster recovery but you'll still need to decide which features to use and when to use them based on your unique requirements for example what constitutes a disaster for your application disaster recovery actions are not trivial so it's important to understand which situations require recovery in the first place and which scenarios make sense to just wait it out what about the other infrastructure were required to run your application if you can readily make your database available in another region will you have application servers load balancers and other resources up and ready to go most disaster recovery centers involve at least some amount of data loss how will you ensure your applications are consistent and how do you decide what is acceptable if the original system becomes available again how will you reintegrate the data that didn't make it to your dr site before the recovery action as with high availability it's important to have a clear understanding of your actual business requirements what are the RPO and RTO for your application what drives those and are they reasonable finally disaster recovery scenarios have many moving parts so it's important to have a test plan and to periodically exercise it with disaster recovery drills AWS and RDS make it easy to spin up complex architectures test them out and delete them paying only for what you use we're going to talk about three major ways that RDS enables disaster recovery first RDS backs up your database to Amazon s3 by taking storage snapshots and copying archived redo logs this allows you to create a new instance at any time either from a snapshot or to a specific point in time second are DEA's Oracle now supports push-button read replicas both in region and cross region replicas are fully managed Oracle active Data Guard standbys and can be promoted in place to become a primary instance finally you can configure logical replication between RDS instances or between RDS and on-premises infrastructure this approach uses third party replication software to extract changes from a source database and apply them to a target database okay let's start with backups there are two ways to back up your RDS instance the easiest is to enable automated backups each day during a 30 minute backup window you specify RDS takes storage snapshots of your database additionally your database is run in archive log mode and archived redo logs are backed up to Amazon s3 every few minutes you may notice that the archive lag target database parameter has been set in your instance this forces the redo log to switch at least once every five minutes allowing RDS to backup your archived redo logs quickly and helps ensure that the gap between the most recent transaction on your instance and what is backed up to s3 which we call the last restorable time stays close to five minutes you can also take a manual backup of your instance at any time even if automated backups are not enabled the only real difference is that manual snapshots are retained until you delete them whereas the retention of automated snapshots is handled by RDS Amazon RDS uses Amazon elastic block store volumes for storage so RDS snapshots are really just a collection of EBS snapshots snapshots are always incremental regardless of whether they are automated or manual so they only backup the blocks that have changed since the last snapshot was taken there may be a brief performance impact during the moment that snapshots are initiated in single AZ configurations this is because IO is briefly paused when we initiate the snapshot this will typically appear as a spike in right latency for a single data point about one second and is why we recommend setting your backup window to an off-peak time for your workload with multi az configurations the snapshot is offloaded to the secondary instance so the effect on latency during the initiation is even smaller in both configurations the actual backing up of changed blocks has no impact on your performance or instance I ops finally snapshots can be copied to other regions or accounts they can also be encrypted or re-encrypted with different Amazon key management service or kms keys restores and RDS always create a new instance you can restore any snapshot and specify things like instance class storage type and parameter and option groups for the new instance RDS creates fresh EBS volumes based on the snapshot provisions in ec2 compute node starts the database and makes it available to you there are a lot of use cases for snapshot restores for example some customers create an instance load up their administrative schemas and jobs then take a snapshot and create all production instances as a restore of that master image snapshots also make it super easy to clone environments and test things like Oracle upgrades and changes to your application you can even write simple shell scripts to do things like restore snapshot and run arm and data file validation on it delete the instance if no problems were found because snapshots can be copied to other regions and accounts they can also be part of your dr strategy the RPO will depend on how frequently you take snapshots and how long it takes them to copy to other regions more frequent snapshots will reduce your RP oh but it will probably still be measured in hours with this solution RTO would be the time it takes to restore snapshot as a new instance verify the configuration and make it available for your applications fortunately RDS makes this very easy to test a few clicks in the console or a single API call will create your new instance from the snapshot you can then test it out and delete it paying only for the time it was running one thing to be aware of when restoring snapshots is that EBS volumes do take time to initialize before reaching full performance when you restore an EBS snapshot the new volume is available almost immediately but the actual contents of the volume are just starting to populate from s3 if you access the block that hasn't been loaded into the volume yet you won't get an error but the read operation will just take longer for a large instance it may take many hours for Layton seas to reach their expected values you can help manage the impact in your application by pre reading your most important tablespaces tables or partitions to ensure their contents have been brought up from s3 liked snapshot restores point in time restores create a new instance however instead of specifying a snapshot to restore you specify a specific moment in time down to the second RDS takes care of identifying the closest snapshot creating the instance and rolling the database forward using archive 3d logs until the recovery time is reached because this requires reading logs it's only available when automated backups are enabled and the restore time has to be in your automated backup retention window this process can take longer if the recovery time requires applying many hours worth of reading logs because of this some customers choose to take additional manual snapshots throughout the day using some sort of automation this gives point and time restore more snapshots to choose from for example if you take snapshots every four hours then a point in time restore would only ever have to replay up to four hour worth of reading logs this is also easy to script and something you want to test carefully as part of disaster recovery drills with point in time restore RPO can be as low as five minutes when using the last restorable time for your instance our tio will depend on the amount of workload it has to be replayed as with snapshot restores this can easily be tested and even automated point in time restore is only available in the same region and account as the source instance so this solution can't help with cross region failures today the second way that RDS helps with disaster recovery is by providing managed read replicas backups provide a great deal of flexibility and are cost effective because you're just paying for cold storage and not running infrastructure but they do take time to restore read replicas on the other hand are up and running and nearly in sync with a master the trade-off here is cost both infrastructure and licensing in order to have an instance that can be pressed into action in the short period of time when you create a read replicas RDS automatically restores a snapshot of your master instance configures it as an oracle data guard standby database and opens it for read-only access the standby is run in maximum performance mode using log writer in asynchronous mode this allows you to have a near real-time replica of your master database without affecting its performance other modes like maximum availability or maximum protection are not currently supported nor are switch over or fill over operations keep in mind that active Data Guard is an Enterprise Edition add-on options so this is only available in the bring your own license model for customers with the necessary licensing in place for high availability purposes read replicas can be multi a-z or not independent of the master database and while read replicas are primarily targeted at read scaling use cases they can be promoted to a primary role and open for read/write access oracle calls this activating the standby database and this splits it off from the original master in its replicas now let's look at some common scenarios to see how these different features interact let's start with a single AZ instance its availability is important to your customers so you enable the multi AZ flag and RDS creates a synchronous replica and monitors the configuration for any failures your application reads more than a writes so you choose to create a read replica to offload those reads to another instance it can be in the same region as the master or in a different region RDS creates the read replica and manages the active Data Guard configuration perhaps the availability of this instance is also important so you enable the multi AZ flag here as well an RDS synchronously replicates its storage to another instance you can also create other read replicas as needed you can even size them differently for example suppose the master instance is running on DBR 5.8 extra-large instances the read activity is relatively light this time of year so you can run the read replicas on DBR 5 4x large perhaps your additional replica is used as a data warehouse so you want to create it as a DVR 5 16x large so that queries can finish quickly and it can be deleted we give you the flexibility to pick the configuration that is right for your workload however do keep in mind that replicas are all fed by the master instance there's not a cascading configuration here so you do need to choose instance types that can keep up with your workload and it's possible that really hot workloads may not be able to sustain 5 replicas RDS Oracle Reid replicas are primarily designed for Reid skilling use cases but they can also fit into disaster recovery scenarios through the promotion feature when you promote a Reid replica replication is finished the database is converted into a primary and open for readwrite access the instance is now a normal RDS Oracle instance and no under re replicas and there's no going back while Reid replicas use asynchronous replication in region replicas are typically only a few seconds behind the master instance and closer to a minute for cross region you can verify this with a replication lag metric in Amazon Cloud watch this means that our po4 replicas promotion is around a minute the promotion process takes a couple of minutes but your RTO would also include any steps to validate the configuration and stand up application servers finally logical replication allows you to replicate some or all of your data to another database this gives you the most flexibility and control over what is replicated with logical replication you'll need another server that hosts the replication software we usually call this the replication instance examples of replication products would be Oracle GoldenGate quest sharePlex or Amazon database migration service and others the replication instance connects to your source database and extracts changes change data from reading logs either using Oracle's log miner API or by reading the binary bits of the reading log itself it then writes these changes to the replication instance as some sort of logical change record did the software knows how to understand finally it takes those logical records converts them into sequel insert update and delete statements and applies them on the target instance with logical replication you can choose to replicate a subset of your data and you can specify transformations to be applied before loading it into the target database you can even replicate to more than one target database and it doesn't have to be Oracle many RDS customers use tools like the database migration service for ongoing replication of data from Oracle to other databases like Postgres or redshift allowing allowing them to perform additional analysis on their data without the license costs of Oracle logical replication can also be bi-directional allowing for active active configurations the drawback of logical replication is that it requires a great deal of expertise and effort to configure and operate RDS doesn't manage replication as part of the service so it's infrastructure that you have to set up and administer yourself it is necessarily an asynchronous process so you should expect RPO to be measured in minutes but typically less than an hour RTO will depend on your exact configuration but can be as low as minutes as well you so what have we learned recovery point objectives and recovery time objective let us evaluate in architecture in terms of expected data loss and time loss when failures occur RDS automates the management of your database infrastructure including complex architectures that support high availability and disaster recovery for workloads with strict availability requirements RDS multi a-z provides a fully managed synchronous replica and automatically recovers from failures with no data loss when it comes to disaster recovery RDS provides a number of ways to get back up and running depending on your business requirements and licensing there's automated backups which let you get to ending back to any moment in time and your retention window read replicas offer a fully managed to Oracle active Data Guard solution for read scaling and can be rapidly promoted to take over if the master is lost and finally logical replication is a powerful and flexible option that gives customers more control over the replication of their data for more details on Amazon RDS for Oracle including documentation case studies videos and more please check out our documentation pages the developer forums are also a great place to check for known issues or get tips from other customers if you're curious about a feature I discussed perhaps you're not sure about how a particular failover scenario is handled or whether you can use a specific functionality of Oracle database with RDS I really encourage you to spin up an instance and give it a shot and see why hundreds of thousands of customers trust already asked to manage their mission mission-critical databases the thing I love about RDS is that I can create an entire new database server or GameDay or restore scenario or test a version upgrade all before my coffee gets cold thanks for spending this time with me and have a great day