An Introduction to Spark for Data Engineering | DataHour by Umesh Kumar

Key Takeaways

foreign now before we dive into the spark architecture over about this part so first we need to understand what is the Big Data like why it came into existence like what was the need for the big data and what was the need for the spark we'll be discussing that so talk about the data the like before which uh start the history so first we need to know what is the Big Data again I'll be taking a very simple definition of Big Data like when we say big data one thing that comes to my mind is the huge amount of data which is in gigabyte terabytes and more than that to like it has basic three pillars variety volume and velocity when we say variety like the big data using the Big Data you can connect to a variety of data sources so that is my variety when we say volume volume is the large amount of data or the humongous data that it can handle or store then the velocity velocity means like it can process the data in a very small amount of time that is why these are the three pillars for the big data now what was the need for the big data so for that first of all we need to go to the uh like the computer so initially like during like initially what happened was during the industrial era when there was a lot of manual work so first thing was computer that came into existence for automation so if you think about the computer like the initial list so the two main operation that it does was first of all it was used for the basic data storage and second was the processing the task so during that data the primary means of storage was the files though like there were there can be any kind of file like it can be Excel it can be CSV or the text file or the word file so data could be stored in the files so after some time like when the data started growing so the companies wanted a central reposition that they have resistant for that relation database was implemented so relational database was better than storing the data in the first in terms so because it allowed data to be stored in the form of rows and columns that is in the structure format it also allowed the data to be stored in the normalized format such as to avoid the data redundancy and to form a land to remove the data duplicacy so this was working perfectly fine and these were most optimized for the read and write operations as well as the transaction processing however with the time like as the time started drawing data started also and it also started growing so at that time there was a need for one more solution like which could deal with the historical data and also could be utilized for the analytical and the reporting process for that data warehousing work uh they were optimal for the historical data and the analytical processing so this was working perfectly fine however with the Advent of Internet so data started growing exponentially why uh why because now a number of the user base for the internet was increased so people from all over the all over the country all over the world started using the internet that is why the data started growing exponentially so at that moment like the database and database these were the only solutions and it was difficult to store and process the large amount of data in a single machine because of the storage limitations