What's New with AWS Storage - AWS Online Tech Talks

all right welcome everyone thanks for coming to our webinar today we're gonna be talking about what's new with AWS storage as of November 2019 my name is Robbie right I'm part of the AWS storage team it's been available us for about three years and so we're gonna be talking about all kinds of things all across the AWS Storage portfolio today and we've got a lot to share so I'm just gonna jump right in so let's start with with one of our most used services Amazon EBS Amazon Elastic block storage as it's also known is a frequently attached to Ames on ec2 instances as as as you can probably guess a block storage platform and as part of that EBS provides a snapshot feature for data protection so for those of you who are familiar with EBS you probably use snapshots today and previously when you were restoring volumes from snapshots that required an asynchronous initialization period of several hours per volume copy and we've made some changes to that with that with some new features and functionality that we've launched recently we call we call this new functionality EBS fast snapshot restore or FS R and with fr you can create up to 10 volumes simultaneously with instant maximum performance from an FS r enabled snapshot so by enabling FS r you get significant improvements in speed so you no longer need to wait for the lazy loading of data from snapshots for a fully performant volumes you get predictability enabling FS r takes about 60 minutes for each one petabyte of snapshot size and after this restored volumes have maximum performance it also helps you lower cost and overhead so there's no need to purchase attach additional ec2 instances to more quickly pulled data blocks from snapshots and then it also gives you the benefit of parallel volume creation so you can create up to ten simultaneous volume restores and some use cases for this functionality our virtual desktop infrastructure backup and recovery back and restore test dev volume copies if you want to create multiple different copies for tests of use cases or booting from custom a.m. eyes and to learn more about this feature you can go to aws.amazon.com slash EBS and now we'll move on to another one of our services this one's a file system we in 2016 we launched Amazon Elastic file system and EFS provides simple scalable fully managed and elastic NFS file system for use with with the AWS cloud services and with on-premises resources the EFS is built to scale on demand without disrupting applications to grow and shrink automatically with files and to eliminate the need to manage capacity EFS is a regional service similar to Amazon s3 and in contrast with EBS which is a single easy service so EFS is being a regional service means that it stores data across multiple AWS availability zones for high availability and durability and recently so the the main things have changed for EFS is just wider availability on November 20th we made EFS avail available in four additional AWS regions the Middle East in Bahrain EU Stockholm South America South Paulo and the Asia Pacific Hong Kong regions and so with the addition of these four regions you can now use EFS in 19 different AWS regions now last year in a wsre event we also launched another file system called Amazon fsx for Windows file server and this provides windows or SMB file system and a fully managed Windows file server instance so over the last few months we've released an extensive list of enterprise ready features for fsx for windows file server that help simplify the lift and shift of Windows applications enable more flexible management and optimize the costs of windows-based file or Windows file based applications and last week we we enhance that by launching several new capabilities we also launched support for SMB transparent failover and and that provides continuously available file shares on multi a ZFS X file systems enabling applications that require non-disruptive access to shared windows data such as sequel server always-on or similar applications and and that enables them to to use fsx as their their file storage and I mentioned multi AZ windows file systems we we actually have now made it much easier for you to create multi AZ windows file systems in fsx and it's important that I've mentioned availability zones a little bit already but I want to I want to sort of unpack that a little bit and explain why that's important so within AWS we have this construct of regions and availability zones and it's different from what a lot of other cloud options will propor so with an AWS we a region is a place in the world where we have multiple availability zones and those availability zones are actually spread as separated by by by typically miles across the metro area and and by spreading them across that area we isolate from local failures things like fires or floods and and and those kinds of things to help increase availability by by giving you that geographic redundancy so within every AWS region you have multiple availability zones that you can replicate across er that you can use to enhance availability of an application so with fsx for windows bao server now you can create multi AZ file systems with a click of a button and and that's a new capability that allows you to span those availability zones without a whole lot of heavy lifting so you no longer need to set up or manage replication across a ZZZ we do that for you and when you create a multi as eval system fsx automatically provisions and maintains an active file server and a hot standby and a hot standby file server in a different a Z each one with their own storage and this capability is is really ideal for for large enterprises with Windows workloads that require shared storage and fault tolerant of fault our own tolerance across a zs this will help customers that need highly available and durable shared windows file storage for things like their home directories their business applications and their web serving environments and like our other storage offerings security is built into fsx from the ground up we-we-we in addition to security we prob'ly provide the ability to support PCI DSS ISO and Sauk compliance and hip-hip eligibility first but coming back to the two security FS x all already offers automatic encryption of data in transit and at rest but now we've added support for fine-grained control of those in transit encryption settings for your file system this gives you the ability to set file system level vs. shared level encryption and the option to enforce in transit encryption to meet your compliance needs we've also added new user quotas and these user quotas will give you the option to monitor and control user level user level storage space consumption so using remote management PowerShell CLI on your filesystem you can you can enable and disable user quotas you can configure quota levels for the for the default and for specific users you can choose how to handle violations and that and and you can choose between either tracking or enforcing by denying further storage space and then you can also list current quota violations so those additions complement the extensive list of security and remote admin features such as direct ad join monitoring and logging API calls using AWS cloud trail and others we've also given you more capability to reduce the cost of Windows workloads so fsx already offers the flexibility to optimize costs based on performance needs by choosing your file system size independently of throughput for your file based Windows workloads we now offer two other cost optimized features for your Windows file systems data deduplication helps reduce the costs associated with duplicated data with typical savings of 50 to 60 percent for general file shares and we now support that that functionality and data deduplication we've also reduced the minimum file size to one tenth of the current level that that we were supporting for SSD based file systems so we've gone gone from 300 gigabytes down to 32 gigabytes and that that is is based on customer requests to enable workloads that have smaller storage size needs so things like file share witness for sequel as sequel server clusters now let's talk about AWS data sync data sync is another one of those services that we launched at AWS reinvent last year and it's a transfer service that makes it easy for you to automate moving data between on-premises storage and Amazon s3 and it's on EFS data sync automatically handles many of the tasks related to data transfers that can slow down migrations or burden your IT operations including running your own instances handling encryption managing scripts networking and data integrity validation data sync is super fast you can you can use data sync to transfer data at speeds up to 10 times faster than open source tools it's automated dancing automates both the management of data transfer processes and the infrastructure required for high performance secure data transfer data sync also includes automatic encryption and data validation and all of that works together to minimize the in-house development and management otherwise needed for fast reliable and secure data transfer data singh helps with one-time migrations or with ongoing transfers so the service is is really built to enable those those one-time migrations recurring processing workflows and automated replication for data protection and recovery and data sync is also super flexible it supports flexible configurations to suit your specific needs including bandwidth throttling copying source permissions and metadata it's integrated with AWS services deeply so that makes it really easy to get started with beta sync all you have to do is deploy the Data Sync agent on-premises connect it to a file system or a storage array select Amazon EFS s3 as your AWS storage and then start moving data and as is important with all of these things it is secure and compliant all of your data transferred via data sync is encrypted in transit and at rest and for each transfer the system performs integrity checks these checks ensuring that the data written to your destination matches the data read from your source validating consistency and they just think also make sure that your data arrives securely intact and ready for use so we've made some improvements with Data Sync we've listened to our customers and and heard a lot of different things so we're announcing a price reduction of 68% for AWS data sync effective from November 1st 2019 data sync has a simple predictable usage-based pricing and with data sync you only pay for the amount of data that you actually transfer and your costs are based on a one point two five cents per gigabyte B that is flat and there are no upfront costs and no minimum fees to use data sync and in addition they sync is now available in five additional AWS regions including Europe Stockholm South America South Holland asia-pacific Hongkong asia-pacific Mumbai and AWS gov cloud US East and with this launch data sync is now available in all commercial and gov regions and finally we've launched test scheduling for AWS data sync that gives you the ability to scale old data transfers so when using data sync you can now schedule data transfers between your NSS servers as a/b servers s3 DFS and and data sync will handle those transfers for you on your schedule now let's take a look at AWS Storage Gateway storage gateway is a hybrid cloud storage service with local caching that gives customers low latency on-premises access to virtually unlimited cloud storage storage gateway supports multiple protocols to enable companies to seamlessly connect on Princeton's applications to AWS storage without having to modify those existing applications it gives you local caching that offloads they're basically offloads to the cloud based on your desired performance parameter so you can fine-tune the balance of latency and scale for your workloads gives you optimized data transfer with bandwidth management automated network resilience and efficient data transfer it's secure and compliant storage gateway benefits from the durability and security embedded in the other AWS cloud storage services and so the storage management tools like versioning cross region replication lifecycle management policies they can they can lower the cost of long-term archiving simplify audit and compliance requirements and safeguard all of your data are available and and at transferable into storage gateway based on the underlying service that they're using storage gateway is also super cost effective so because cloud storage is delivered and billed on demand you you always have just the right amount so workloads can expand and contract or back up an archive storage can expand without upfront media costs and you can provision additional storage capacity without to provision new hardware and finally storage gateways are all are really easily managed via the AWS console and most importantly once your data is in the AWS cloud it can be queried accessed and used across the entire portfolio of AWS applications and services so we've had we've launched several new things for storage gateway in the last week we've four workloads that that require under uninterrupted operations and performance SLA x' customers have asked for a highly highly available solution that's resilient to hardware storage networking or software errors and their on-premises infrastructure customers are looking to simplify their operational processes of monitoring and optimizing resources allocated to their to their gateway VM to align with increasing performance needs and for workloads that can't be interrupted customers want to align software updates on their gateway with maintenance windows and their IT policies and now let's jump over to Amazon s3 so s3 customers receive a really high level of availability and durability for their s3 day and every AWS region customers often have compliance or business requirements to replicate their data to a second region hundreds or even thousands of miles away from their primary region and so you know s3 is is well known in the market and there are literally billions of customers that use it every day and as we talked about earlier you have these constructs of regions and availability zones and and like Amazon EFS that we talked about earlier s3 is a regional service and so every time that you write into s3 you have the ability to or you have you have Geographic replication already built-in because we're writing across three geographically separate availability zones within an AWS region and so we've launched we want some new capabilities around replication particularly you know we well we've we've had for a while cross region replication that ability to replicate data from any AWS region to any other AWS region and replication provides an automatic mechanism to make identical copies of those objects in a destination of your choosing so you can you can do that for multiple different use cases whether you want to have data locally for users in a in a different part of the world to reduce latency or if you need to have a protection copy beyond what you get from the regional protection of the geographic protection that you have built into every AWS region so replication can can help you meet some of those data resiliency and compliance requirements or place place copies of your data close to internal users for that operational efficiency I talked about earlier in those youth cases involve replication across regions other needs involve replication within the same AWS region for example replication can be used to replicate data between your production and test accounts helping your teams be more agile or to simplify analytics jobs by aggregating logs from multiple services in different buckets into a single bucket for analysis finally a replication can also help you protect against unintentional deletes by keeping a copy of your data in a different account often in a less expensive infrequent access storage class that all allows you to meet data residency requirements that require data to stay within a particular AWS region and not replicate to across regions to another AWS region and so let's review how rs.3 replication helps you meet those needs next s3 replication is an automatic and asynchronous process that makes an identical copy of your object retaining all the metadata that's on the source object including the the object creation date and time and version IDs s3 replication has always worked across AWS regions so customers have asked us to extend that capability to replicate within the same region as I mentioned earlier and so we've recently announced exactly that and that's and and so and now you have the choice to choose the destination bucket in the same AWS region as the source with an s3 so let's take a closer look at those capabilities so s3 replication gives you the choice of replicating entire buckets a set of object objects under given prefix or specific objects that you identify with object tags of your choice replication begins as soon as you upload an object to s3 that's covered by a replication policy making an identical copy of that object to a destination bucket of your choice once you've selected your source dataset you select a destination bucket which as we've mentioned can now include buckets in the same region as your source bucket to enhance data security you can choose to change the ownership of your replicas copies and also replicate them to another account so that no single user or account can erroneously or unintentionally delete all copies of your data and depending on your use case for the replicas copy you can choose to keep in a frequent access storage class or replicate it directly into and an infrequent access or archive storage class such as s3 glacier s3 glacier deep archive you can also specify different lifecycle policies on your source and replica objects to optimize cost based on your access patterns in both the primary and secondary locations so since the launch of crr in 2015 s3 customers have replicated trillions of objects and exabytes of data and today we're making this robust capability even more count even more powerful with a new feature called s3 replication time control customers have been asking for details on replication time and metrics for compliance requirements and other use cases for some time now and we're pleased to announce that s3 replication time control is now generally available in all AWS commercial regions replication time control provides customers that are often interested in the the time it it takes for objects to replicate to their destination which is influenced by the object size the count and other factors with visibility and additional control over replication time you can now use all of the capabilities of s3 replication with a new 15 minute replication time service level agreement and a set of cloud watch metrics and event notifications that are part of s3 replication time control s3 replication time control gives you a set of metrics that you can use to monitor replication for each replication rule these metrics tell you how many objects and bytes are pending replication and the maximum replication latency of the objects covered by your replication time control policy you can view these metrics in the management console or in cloud launch and set up alarms to notify you when any of these metrics exceed a threshold of your choice these cloud watch metrics are aggregated for each replication rule and you may also want the want to to monitor a replication at the object level which you can do with the s3 replication events that we'll review next you can do you view these metrics in the s3 management console or in cloud watch and set up alarms to notify you when any of these metrics exceed a threshold of your choice these cloud watch metrics are aggregated for each replication rule replication time control gives you predictable replication times that most objects replicate in seconds 99% of objects replicate within 5 minutes and 99.99% of objects replicate within 15 minutes this replication performance is backed by an SLA that covers the replication of 99.9 percent of objects in each replication region pair in any billing month in the unlikely case that the SLA is not met you will receive a credit on the replication time control charges data transfer and request request charges and the destination storage charge is associated with the objects that did not replicate within 15 minutes in the billing period effective you can view the full service level agreement on AWS SLA website and with that we will open the floor up to questions and answers on any of these recent launches