Optimizing storage costs using Amazon S3 - AWS Online Tech Talks

Key Takeaways

[Music] hello welcome to today's presentation optimizing storage costs using amazon s3 i want to start by saying thanks to those who are joining us today my name is andrew cutscene and i'm joined today by jesse felix jesse and i are product managers with amazon s3 and today we'll be talking about s3 storage classes how customers with different patterns of cost optimization use amazon s3 to optimize your storage costs we'll then review how you can use s3 storage lens to analyze and act on your storage and storage costs and finally we'll double click and dive deep into the s3 intelligent tiering storage class which delivers automatic storage cost savings before we get started i want to say thank you and thanks to the many millions of customers who use s3 today and help us innovate what's particularly fascinating about this image is really the breadth and the depth of storage use cases and how these use cases have evolved over time when amazon s3 launched in 2006 one of the top use cases was website hosting but today we see customers of all sizes and across every industry running every imaginable workload on amazon s3 these use cases span from running data lakes to storing data that is used for machine learning analytics content delivery and medical research each of these use cases have unique access patterns and later in today's presentation we'll talk about how you can identify the access patterns of your storage the reason why so many customers across every industry use amazon s3 is because of its industry-leading performance it's easy to use management features to manage your data and s3 is the only cloud storage service that lets you manage your data at the object bucket and account level for example you can use s3 storage lens to view your object storage usage with drill downs to the account bucket and prefix level s3 delivers the broadest data movement storage options and the widest range of cost optimization capabilities including s3 glacier deep archive which delivers the lowest cost storage in the cloud and lastly s3 provides industry-leading scalability availability and durability however the topic of today's webinar is optimizing cost so let's dive into that building on top of s3's industry leading scalability availability security and performance i'm super excited about today's presentation because i get to talk to all of you about some of the ways that customers can redefine cost optimization in s3 specifically i want to highlight two recent announcements that i think redefine how customers can optimize their storage cost first at re invent we announce s3 glacier instant retrieval a new storage class that delivers the lowest cost storage in the cloud for long-lived rarely accessed data that requires milliseconds retrieval this storage class provides the same latency and throughput performance as the s3 standard and the s3 standard infrequent access storage classes and like s3 standard infrequent access it's designed for 99.9 availability and 11 9's of data durability by redundantly storing data across a minimum of three physically separated aws availability zones and it's really easy to get started with s3 glacier instant retrieval you can upload data directly to the storage class or you can set lifecycle policies to transition objects from s3 standard or s3 standard infrequent access to s3 glacier instant retrieval second with the launch of s3 glacier instant retrieval we also introduced the new axis tier and s3 intelligent tiering s3 intelligent tiering is a storage class that delivers automatic storage cost savings by monitoring access patterns at the granular object level and automatically moving objects that have not been accessed to lower cost access tiers it's really the ideal storage class for data that has unknown changing or unpredictable access patterns and so if you need instant access to data and have unknown or changing access patterns you can receive the same storage price as s3 glacier instant retrieval with this new axis tier and s3 intelligent tiering just in the same ways that customers have evolved their workloads over time customers have evolved in the ways that they optimize their storage costs so now i want to take a step back and talk a little bit about the common patterns of cost optimization that we see in amazon s3 the first pattern we see is data with known or predictable access patterns do you have data that becomes infrequently accessed after a definitive period of time let's take for example user generated content like videos and photos that we share with our friends and family that content is frequently accessed right after we upload it but becomes infrequently accessed a few weeks or perhaps even a few days later for use cases like these many customers know when data becomes infrequently accessed or can usually pinpoint the right time they should move data to a lower cost storage class that is really optimized for infrequent or archived access of storage but what if you have data with unknown or changing access patterns for use cases like storing shared data sets where data is accessed by several different applications it's really common that many users within an organization will access s3 with different tools for example amazon athena makes it really easy to analyze data in s3 using standard sql or they may use amazon regis spectrum to run queries against petabytes of data in s3 without having to load or transform any data the access patterns for these use cases are highly variable over the course of the year and can range from little to no access to data being read multiple times in a single month or even in a single day which could result in high retrieval charges if stored in an infrequent access or archive access storage class some examples of use cases with known or predictable access patterns are medical images typically healthcare providers expect that an x-ray or a ct study is frequently accessed for a short period of time but then becomes rarely accessed after a few weeks another example includes learning resources over the last couple of years we've seen explosive growth in content from education institutions and universities it's really common for this content to be frequently accessed by students over the course of a semester but over time this content becomes more rarely accessed lastly and perhaps a more relatable example is user generated content i'm sure many of us here today use a photo sharing website for storing personal photographs that we expect to treasure forever once we upload a photograph we may share it and it would then get downloaded a few times but after a period of time those images likely see a major decline in access frequency but i'd say that the vast majority of data has unknown changing or unpredictable access patterns i think that use cases like data lakes and analytics are perfect examples of data that has unpredictable access patterns where many users and applications are accessing the same underlying data let's take another example of public research data that is available to scientists and professors including weather driving economic data oftentimes it's very common that these data sets are infrequently accessed and even rarely accessed but unforeseen world events can drive a surge in demand in historical economic or weather pattern data now let's go through the broader range of s3 storage classes it's really important to choose a storage class that best matches your workload requirements and how you want to use your data so in order to support these patterns of cost optimization we just discussed s3 has a number of storage costs to choose from and for every imaginable use case in the following we're going to talk about how you can match your access pattern whether it be known or unknown to the right s3 storage class since the launch of amazon s3 in 2006 we've not only decreased storage pricing but we've accelerated innovation by introducing new storage classes so that customers can optimize their cost based on the requirements of their workload in 2019 we introduced s3 glacier deep archive which offers the lowest cost storage in the cloud in our re invent we introduced the esri glacier incident retrieval storage class which delivers the fastest access to low cost archive storage we also launched s3 intelligent hearing in 2018 the only storage class in the cloud that delivers automatic storage cost savings based on the changing axis patterns of your workload let's talk about what this all means because the vast majority of data has unknown or changing access patterns many customers use s3 intelligent hearing to automatically optimize your storage costs s3 intelligent tiering is the only cloud storage class that delivers automatic storage cost savings when data access patterns change without performance impact or operational overhead every event we announce a new access tier that is designed for rarely access data that needs millisecond performance customers use s3 intelligent tiering to store shared data sets where data is aggregated and accessed by different applications teams and individuals whether for analytics machine learning real-time monitoring or other data lake use cases and i want to highlight that s3 intelligent tiering is really becoming the standard or the default for many customers especially those building data lakes on amazon s3 and for data with known or predictable access patterns customers can choose from a set of storage classes that are cost optimized for different data access patterns s3 standard is the ideal storage class for frequently accessed data this is the best choice if you access data more than once a month s3 standard is also the ideal storage class for small short-lived objects s3 standard infrequent access in the storage classes to follow are really designed for less frequently accessed workloads where your cost to store data decreases but the cost to access your data moderately increases s3 standard infrequent access is the ideal storage class for data that is accessed once every month or two s3 glacier instant retrieval is the ideal storage class for data that is accessed once a quarter and for archive data that does not require immediate access s3 glacier flexible retrieval delivers additional cost savings with retrieval times of minutes to hours an s3 glacier deep archive has the lowest cost storage in the cloud with data retrieval in hours additionally amazon s3 offers storage classes for specialized workloads for easily recreatable data or for storing secondary copies in a different aws region you can use the s3 one zone infrequent access storage class which delivers additional cost savings over s3 standard infrequent access but stores data in a single aws availability zone and lastly s3 on outpost delivers object storage in your on-premises environment with the s3 api and capabilities that you use in aws today to recap in evaluating when you should use the lower cost storage classes designed for less frequently accessed workloads with milliseconds access requirements there are two factors to consider the first is the frequency of access and the second is the duration of storage for example s3 glacier instant retrieval is the ideal storage class if you access data once per quarter this is because although the storage price is lower than s3 standard infrequent access the cost to access data is higher meaning that there's a breakeven point where if you're storing data that is accessed too frequently it makes sense to keep that in s3 standard infrequent access additionally s3 glacier instant retrieval has a minimum storage duration of 90 days so let's say you upload a file you expect to delete within 90 days we recommend that you choose the s3 standard or the s3 standard infrequent access storage classes keep in mind here that s3 standard infrequent access has a minimum storage duration of 30 days there are a few very simple ways to upload data to one of the storage classes that is cost optimized for infrequent or rare access of storage in this example i show you how to upload an object directly to s3 glacier instant retrieval here you can specify glacier underscore ir and the put api request header to write objects directly to the storage class customers should upload directly to s3 glacier instant retrieval if they know that their data is rarely accessed right at ingest or you can use s3 lifecycle policies to move data with known or predictable access patterns to lower cost storage classes over time lifecycle policies are configured based on the creation date of the object and can be filtered to apply to the whole bucket a prefix or a set of tagged objects remember that s3 standard s3 standard infrequent access and s3 glacier instant retrieval deliver milliseconds access to storage and the s3 glacier flexible retrieval and esri glacier deep archive storage classes are designed for asynchronous access to storage with retrieval times of minutes to hours when choosing which objects to lifecycle to lower cost storage classes or when organizing your data in general it's also important to think about the object sizes of your data that's because as object size increases for a given amount of storage there's a decrease in request cost so we recommend that customers batch small objects into large objects to reduce the total number of requests and therefore decrease the request cost this is particularly helpful if you have millions of small files for example if you're ingesting objects to s3 via kinesis firehose you can set threshold for object batching before uploading to amazon s3 in addition at re invent we also introduced the ability to filter lifecycle rules based on object size for example during life cycle transitions you can optimize your cost by only moving objects larger than a specified size to storage classes like s regard fish or deep archive and in addition you can save cost by controlling the number of non-current versions you have for a version bucket to recap bicycle policies are rules you can set up to move objects to another storage class after a given number of days these rules are based on the creation date of the object and can be filtered to apply to the whole bucket a prefix or tagged objects let's take for example a typical medical imaging workload when an x-ray or a ct study is generated it's typically frequently accessed for a short period of time by the radiologist the practicing physician and the technicians after a period of time that x-ray or ct study is likely rarely ever accessed again perhaps for a patient visit or in the event that a radiologist needs to review a patient's prior history but it must be immediately accessible when requested in this particular scenario you can life cycle your data from estuary standard a storage class designed for frequently accessed data to s3 glacier instant retrieval after 90 days and then after an additional period of time life cycle your data to s3 glacier deep archive to further optimize your storage cost next i'll hand the mic off to jesse who will talk about using s3 storage lens to analyze and act on your storage and then dive deep into the s3 intelligent hearing storage class thank you andrew to reintroduce myself my name is jessica felix a product manager here at s3 up until this point we've talked about how customers with different patterns of cost optimization manage a storage footprint on s3 and you might be asking how do you identify whether you have a a workload that has changing or predictable access patterns to help answer that we're going to dive into how sd storage lens can help you analyze your storage data access patterns to help you choosing the right storage class storage lens is an interactive highly visual dashboard built into the s3 console which is auto configured for all customers for free storage lens provides 29 usage and activity metrics that are pre-aggregated without you needing to do any additional data processing alongside these metrics storage lens provides callouts to identify best practices to improve cost efficiency and data protection to provide you a little bit more context why we built storage lens we wanted to give you organization-wide visibility into usage and activity metrics the ability to analyze data at the granularity that you need for root cause analysis if certain buckets are growing faster than expected also provide you with meaningful insights that you can quickly action for example identifying buckets that have not had any retrievals for months at a time where you should be using s3 glacier instant retrieval or even a glacier deep archive many customers today are actually using s3 storage lens advanced metrics to track activity metrics over time helping them understand accounts or buckets with data that have not been accessed for long periods of time oftentimes customers that have predictable access patterns observe that the percentage of data accessed each month we call this the retrieval ratio is consistently low after data becomes rarely accessed and based on these insights you can configure an s3 lifecycle policy to transition this data into glacier instant retrieval if you need immediate access when data is requested or even glacial deep archive to get the lowest storage cost in the cloud alternatively many customers observe that their data can become cold for a period of time but often data retrievals spike unexpectedly over the year and this is because many customers use s3 to store shared data sets or content that can unexpectedly become frequently accessed and for use cases like these where access patterns can change over time customers use the s3 intelligent tiering storage class because it is designed to optimize storage cost by automatic automatically moving data to the most cost effective access tier when access patterns change we'll talk more about intelligent tiering in our next section i also want to call attention to the fact that many customers today use the in-console experience for storage lens but we've gotten feedback that customers want to integrate storage lens metrics into their own tools and for that reason we integrated s3 storage lens with amazon cloud watch so that customers can consume sd storage lens metrics in the tools that they prefer specifically now with the new cloudwatch capabilities unlocked for storage lens metrics you get a unified view across multiple aws services in a single dashboard you can also configure alarms to trigger sns notifications on s3 storage lens metrics and finally through cloudwatch you're able to access your metrics via api so you can build custom applications let's dive into an example of what that looks like today many customers starting you started using cloudwatch anomaly detection with their s3 storage lens metrics anomaly detection is just another method to set alarms but instead of specifying your own threshold you let cloudwatch analyze historical metrics using statistical models to come up with a normal baseline range and then cloudwatch will alert any time that a metric value falls outside of that range we're really excited about what we hear customers doing with s3 storage lens to analyze their data usage and activity usage patterns ultimately enabling them to optimize your storage cost now s3 storage lens like we discussed is a powerful tool to get a detailed understanding of your data usage patterns but what if you have unknown or changing access patterns or you want an easy button to automatically optimize costs without needing to do any analysis the answer to that question is s3 intelligent tiering the only storage class in the world that optimizes storage costs automatically as data access patterns change in fact since the launch of s3 intelligent tiering back in 2018 we see that storage cost savings to our customers exceed 250 million dollars and that number continues to grow every single day and when we look at that number here at s3 we actually get really really excited about it and that's because we know that passing on this value to you helps you grow your business and deliver new experiences to your customers so let's talk more about what the s3 intelligent tiering class is how it works and when exactly should you use it s3 intelligent tiering is the only cloud storage that delivers automatic storage cost savings when your data access patterns change with the new archive instant access tier intelligent tiering now automatically moves objects between three axis tiers without any impact on performance and without any operational overhead the new archive instant access tier delivers 68 lower storage costs for data that becomes rarely accessed compared to the infrequent access tier and again there's no operational overhead no lifecycle fees for moving data between access tiers and no retrieval fees when using intelligent tiering all of this while providing the highest durability and the highest availability earlier we talked a lot about optimizing costs for workloads with known or predictable access patterns but in fact the vast majority of data that we see in s3 has unknown or changing or up unpredictable access patterns and that's because we can identify that a certain data set is rarely accessed at a specific point in time but we just don't know when access patterns for that data will change in the future and for these use cases you should use the s3 intelligent tiering storage class to automatically optimize storage costs based on the changing access patterns of those data sets and that includes optional archiving capabilities to get the lowest storage cost for data that's not accessed for very long periods of time let's talk a little bit more about how the intelligent tiering class actually works in exchange for a small per object monitoring and automation charge intelligent tiering monitors the access of every single object uploaded or transitioned into the intelligent hearing storage class previously any objects not accessed for 30 consecutive days were automatically moved to the infrequent access tier whenever that object was not accessed for 30 consecutive days if access later it would simply move back to the frequent access tier without any retrieval charges or any impact on performance and now with the launch of the archive instant access tier we give customers the benefits of automatic cost optimization and the lowest storage cost for rarely accessed data that needs immediate retrieval similar to today objects uploaded to the s3 intelligent tiering storage class are first stored in the frequent access tier and any objects not accessed for 30 consecutive days are automatically moved to the infrequent access tier at 40 percent lower storage cost and now intelligent tiering automatically delivers 68 lower storage cost relative to the infrequent access tier for any objects not access for a total of 90 consecutive days following the launch of the archive instant access tier we have heard from hundreds of customers that they have achieved their cost optimization goals for the year this is really exciting because we're hearing customers are able to invest in new projects that they didn't think that they would be able to invest in a little more why we built the archive instant access tier many of our customers today told us that they have petabytes of data that become rarely accessed for periods of for periods at a time but they need millisecond access when data access patterns change in the future to provide you with an example we have many customers that are storing catalogs of music assets that always have to be immediately available when requested by a wide range of streaming providers to recap the sr intelligent turing storage class now automatically optimizes between three axis tiers with the new archive instant access tier delivering 68 lower cost than the infrequent access tier there's absolutely no impact on performance and nothing that you need to do to benefit the durability and availability of intelligent tiering is the same with eleven nines of durability and three nines of availability in addition i also want to call attention to two changes and optimizations that are delivering additional cost savings for our customers first any objects that are smaller than 128 kilobytes are not eligible for auditoring and are no longer monitored or charged second intelligent tiering no longer has a minimum storage duration period and what that means is that any objects that you delete within 30 days will no longer incur any early delay charges and the reason why we did this is because many of our customers told us that they want to use s3 intelligent tiering as their default storage class without needing to analyze their object size or object life distribution of their data and with the increasing number of customers using s3 intelligent tiering as their default storage class customers want to know what the most cost effective way is to store that data and the most cost effective way to uploading data to s3 intelligent tiering is by modifying your put api request header once you do this any newly created objects will automatically be uploaded to the s3 intelligent tiering storage class next let's talk about how you can achieve the maximum storage cost savings for data that is not accessed for very long periods of time to give you some context we continue to observe that the volume of data growth continues to accelerate and with that growth customers want to control the storage cost in order to do this you have to have an archiving strategy in place and today we have millions of customers that are using amazon s3 glacier storage classes which are purpose built for data archiving the amazon s3 glacier storage classes provides you with the highest performance most retrieval flexibility and the lowest cost archive storage in the cloud and to combine the benefits of automatic storage cost optimization with the lowest cost of archiving customers can use the s3 intelligent hearing source storage class let's talk more about the amazon s3 glacier storage classes for objects that you know are called unengis such as financial records that you need to store for compliance reasons in the case of an audit or data that you know will be rarely accessed for the foreseeable future you can choose from three archive storage classes optimized for different access patterns and storage durations for archive data that needs immediate access such as medical images and news media assets choose the s3 glacier instant retrieval storage class for data that does not require immediate access but you need the flexibility to retrieve large amounts of data sets at absolutely no cost such as backup or disaster recovery use s3 glacier flexible retrieval and to save even more on long live archive storage such as compliance archives and digital media preservation choose s3 glacier deep archive the lowest cost storage in the cloud with data retrievals within 12 hours and what if you have unknown or changing access patterns for customers that have data with changing access patterns or customers that want to make sure that they're archiving the data that has not been accessed for a very long time customers use s3 intelligent tiering in fact we observe a growing trend where customers want to automatically archive based on the access of data as a result more and more customers are activating the deep archive access tier within the s3 intelligent tiering storage class to achieve the lowest storage cost savings for data that hasn't been accessed for a year or even longer the asynchronous archive access tiers are ideal to automatically identify and optimize storage cost for objects that are not accessed for long periods of time both the archive access and the deep archive access tiers are optional because data retrievals are within minutes two hours you can choose one or both of the asynchronous access tiers depending on your application requirements and you can extend the time to move data to the asynchronous archive tiers by up to two years and once you turn on the asynchronous archive access tiers and s3 intelligent tiering you can also get automatic notifications to track when objects are moved to the archive access or the deep archive access tier for example a lot of our customers in the media and entertainment industry they need to update their media database automatically to know which assets are immediately accessible when requested by their end users once our end users make a request for a tv recording or very rarely access media asset that has been archived their application knows that it first has to be restored and it notifies our end users that they that they will have to wait several minutes to hours to close our presentation we want to leave you with a few key takeaways first use s3 storage lens to analyze your storage access patterns this can be really useful in analyzing activity metrics over time to help identify whether you have a workload that has predictable or dynamic access patterns use our building blocks like s3 lifecycle for data with known or predictable access patterns and use s3 intelligent tiering by default for data with unknown or changing access patterns finally put in place an archiving storage strategy using the amazon s3 glacier storage classes this concludes our session today and we want to thank you for your time [Music] you