AWS Databases vs. Storage: Choosing the Right Service for Data Science

Key Takeaways

Now what do you mean by databases? If that is a question I'm going to ask to all of you. Maybe I'm going to get a definition. Database is the collection of data stored in rows and column in a tabular form. It's very common. That's a very common definition we used to hear from our school days. Okay? And we used to feel this definition as a right definition all the time. But let me make this very clear to all of you. These days in the current trend, we are not only dealing with structured data. We're not only dealing with data stored in rows and column in a table form. Instead, we can deal with any type of data or any kind of data coming from any of the data source. Okay. So in a similar manner if you have to talk about databases you can simply say databases are used to organize query and manage data efficiently. Now why AWS databases? The first thing about AWS databases is most of the administrative task will be taken care by AWS based on the choice of service you're making. Okay. and then databases offers a high performance, secure and reliable foundation to power generative AI solutions. Okay. So these days when we are talking about or when we are living in the world of generating AI, we are also looking towards the services like from the generative AI point of view. Okay. But other than generative AI, these services can help you out with different features and these are widely used in the industry even before generative AI or recent generative AI. I can say that. Okay. So best part what I like about this document apart from benefits and databases if you will scroll down they are talking about many more things. Okay. And these are the database services. Now this table is very useful for the candidates who are not aware about different type of databases. Okay. Now obviously this would not be an exam question. Okay. But the idea is there are different categories which has been used in order to divide AWS services like relational key value in memory document and graph and even white column and time series. Okay. And they have given you first of all what are these database types. So if somebody is new, if somebody wants to get a basic understanding don't want to get a deep dive into they can refer this particular article. Okay. Then we are talking about the use cases for each database type and what are the services supports these use cases or what are the services supported by these database types. Okay. Anything you can say the major categorization of databases like we can do that is relational and non- relational. Relational means where the schema or the structure is defined. Non relational means where the structure or schema is not defined. Okay. Now as a part of our videos lecture series this particular video lecture series we will be talking about RDS and ratchet the two major database categories. Okay. So you can see from here like what do you mean by relational? Relational databases are having predefined schema and relationship between the data. Okay. Key value these are very commonly optimized for the storing and retrieving large amount of data. Okay. So here the power is database delivers quick response time even in extreme volumes. Okay. Of concurrent request in memory means you want to reduce the overhead. you want to reduce the burden on the master database or main database by providing the data or by responding to the queries from the memory. Okay. So they are used for the applications that require realtime access to data. In a similar manner we have document graph and other database types. Okay. Now before I'm going to conclude this particular video lecture, let me ask you what are the key takeaways okay from our discussion till now. So key takeaways means storage service you can assume where and how you keep your stuff. Databases you can say how you structure and retrieve that stuff efficiently. Means first of all you are storing somewhere in the storage service. then you are basically defining a structure. You are retrieving the data efficiently. Okay. So if you're going to compare with the real life in the real life you won't store tax records in your backpack or keep thousand of books on a whiteboard. Each service fits a specific role. And when you are going to explore a particular service category, you will be able to understand the major benefits these services leverages. Okay. Now I also want to discuss about one last question with all of you. Okay. My last question is what is the difference between storage and databases? Very important question. Okay. Can storage act as a databases? So answer to this question whenever I have to talk about see actually you can provide lot of distinctions you can provide some similarities in storage and databases one very simple answer which I always give that is storage are used for long-term storage databases are used for performing day-to-day queries or day-to-day operations or daily activities okay so storage purpose is to store, retrieve raw data and database purpose is to organize and query and manage data efficiently. Okay. Then another difference can come with respect to the data type. So storage options which we have seen these storage options supports structured, unstructured and semistructured data. Okay. In databases structured data is stored in tabular form in rows and columns. If we are talking about semistructured, we store the data in key value and JSON format. Okay. So on the same line, there can be many differences between storage and databases. But yes, you can use both of them together. As I had given one example to all of you, I'm going to give it again. So like in your real life, you would not store tax records in your backpack or keep thousands of books on a whiteboard. Each service fits a specific role. Okay. So if there are similarities, understand the similarities but on the other side don't deny the differences among different AWS services and that is a reason they have been designed for different use cases. Okay. So in this video lecture we are going to talk about deep dive into S3 for data science workflows. We will be covering different concepts here that includes folders, versioning and life cycle policies. I have taken a blog for all of you and I am also going to help you out with the help of some diagrams. Now first question which comes to our mind why S3 is central to data science workflows like whenever we are exploring AWS why we want to talk about S3 all the time. Okay. So, S3 is a data lake backbone. Whenever we talk about data lake, we can call data leak as a single source of truth or we can call data lake as a centralized repository where we are storing data in its native form or raw form. Okay. Now S3 being a powerful service or cost effective storage option it gives you durability means chances of data loss is minimal as AWS creates the copies of object in multiple availability zone. Obviously there is one exception as per the storage class. Okay. It is scalable and integrates tightly with the AWS services and machine learning tools. So if I have to talk about some use cases data scientists typically use S3 for the first one is raw data injection and storage. Then you have intermediate processing output. You can use S3 for model training data. You can use S3 for model artifacts like checkpoints, logs, metrics and you can use S3 for the final reports or dashboards. It's your choice what exactly your use case is. Okay. Now let me first of all give you an idea from the management console. So if I'm going to take you all to the management console, you know this is our landing page which is also known as console home. Means you know that if you have to start exploring AWS services, you need to have an active AWS account. And whenever you have to explore a specific service, you can search the name of the service in the search bar. Okay. So we had explored S3 in our past video lecture series. Also we know identifier for S3 is bucket. Okay. You can relate bucket as a container or a repository where you are storing your data and data is stored in S3 in form of object. Each object is characterized by three things. One is data, another is metadata and third one is object key. Okay, means actual content. Then the info about that content and finally the object key as an identifier.