An Introduction to Big Data Processing using Apache Spark | DataHour by Akshay Chauhan

Key Takeaways

okay we are all set uh what you upside the virtual stage is all yours now thank you very much uh first of all I would like to thank um analytics video for organizing um data our series it's really a good platform um to uh Apostle um various peoples on the technical uh data our session processes and the different Technologies so today we are going to deep have a deep dive over um Apache spark so we will try to aim to understand what the purchase part is at the same time we will also see how a purchase path is up using valid Computing and distributed computing applications for the optimal and uh processing of data so I would like to share my screen now just give me a couple of moments I hope you are able to see my screen so let's take off so in this session we will focus on introduction to big data processing using Apache spark and that's a brief slide about me um so I have around 10 years of experience in data engineering and in the past have worked in the various domains of Finance automobile Telecom internet and Retail so I did my graduation computer science and engineering and after that I had an MBA from IIT Delhi and currently working as working as a research scholar at um Lucknow so I did other uh some subscribers in the past and some of the journals on data processing and natural language processing and I also contributed towards the technical review of some of the books speaking in terms of core core interests for me data engineering is one of the area where I'm very passionate about and apart from that I love to read and of course travel to explore different cultures so that's a very brief uh background about me okay so let's dive into what's the agenda for today's session so we will have a conversation about uh evolution of data pipeline how the data has evolved across the different years in the past and how the spark has played its role in the Journey of the data pipelines processes then we will have a brief introduction to a purchase path where we focus upon what the Apache spark is and how the aperture spark has evolved across the different years in the past after that we will briefly touch upon the spark ecosystem how the Apache spark ecosystem Works what are the different components of the Apache spark ecosystems are and what are the offerings which Apache spark ecosystem provides for us after that we will deep dive on the Distributing processing in a spark where we will focus on understanding uh how the actual distributed processing happens across the different parts of the nodes within the cluster and then we'll focus more on the logical and the physical processing uh executions and optimizations of codes that happens behind the screens in this part we will also focus on few bets on the spark terminologies which you may come across very frequently so for example what the spark jobs are or what how the jobs are broken down into stages and what the tasks are and some of the integrities about the various terminologies for example Transformations actions rdds data sets so we will just try to focus and clear up some doubts over those terminologies then we'll also focus on some of the spark Performance Tuning in our tips uh where we how can we improve our the spark executions and also we'll try to understand how can we gain a more efficient spark computation well applying during the cluster and then we will have some kind of exercise using the Google collab and in the end we will have uh maybe if the time permits then we might take questions um so you feel free to put your questions in the chat chat box and then we will take it up from there cool so setting up an agenda then let's take off um today to the session so let me just move it the sideways Okay cool so evolution of data pipelines why it's important uh to understand the evolution of data pipeline across different uh years or in the past the reason is That's How uh it has shaped the current state of the Apache spark as well as other different technologies that are available for the data processing in the current ERA so if we uh please if you go back to the year 2004 uh then Jeff and Sanjay came up with the concept of mapreduce processings which allowed us to not order only process the data at the scale but also lays the foundation of MPP systems or if we talk in terms of uh efficient processing and computations then it allowed us to you know break down all the computation across the different worker nodes and then aggregated the results up to the driver nodes but the challenge in this map reduce concept was that every time you are breaking down and aggregating the intermediate results the aggregated uh results in the intermediate steps are being written into the storage systems which causes an inefficient IO computations and IU computations are always expensive to process the and cause becomes a bottleneck in the data processing but nonetheless it provided a very good uh foundation in the framework how can we distribute uh the jobs across the different nodes and how can we just go and implement the horizontal scaling or further data processing after that uh in the year 2006 when a duck came up with um the concept of Hadoop ecosystem uh in the Hadoop ecosystem he practically used mapreduce as an underlying data processing framework but at the same time he also introduced some of the other components of the Hadoop ecosystems so for example if you can imagine if you're working on with a Hadoop cluster then you will need a cluster manager you will also need um file system where you will be actually processing and storing your data so he introduced Apache and um and Hadoop as a cluster manager and then you have a very famous hdfs components which came up uh during this Hadoop ecosystem implementation but still at this particular Point uh implementing mapreduce uh Concepts and writing the custom Java codes is a bottleneck and it's very cumbersome for the developers who are involved in big data processing so in 2007 Chris came up with the cascading um as a abstraction layer which takes performed some of the simplifications of writing the mapreduce of but it was still um not that but easy to write a mapreduce jobs and then performs The Parallel processing across the different components into the system then in 2008 to simplify the data processing uh we had an Apache pick that was um introduced which it's a high level platform for building Hadoop jobs and writing out your bad jobs and productive processing the data and the batch format but still it was cumbersome to write the mapreduce components within the Hadoop equal system so until this time you can see the trend uh that we also wanted to process the data at a scale in a very efficient manner but at the same time we also wanted to simplify how we process the data so the focus was not only on the efficient data processing but also simplifying the data processing operations we had different technologies that were being introduced at this point and also the volume of data was growing so the emphasis on both the data processing efficiencies as well as the simplification of data processing uh in 2011 when we had the use cases of streaming data or the near real-time data processing then Kafka came into the picture and it was introduced by Cherry while working at LinkedIn and in the same year Nathan also came up with Apache storm which provided and distributed real-time computation engine for processing data streams so up to uh 2011 you can see we had certain Technologies to process the data and the batch format at the same time we also had some technologies to process um the data in the features in form of real-time streaming streams so in 2014 at this particular Point uh spark1.0.4 was released and which provided a spark SQL as an abstraction layer on top of spark so why this um uh development was a very significant in 2014 because SQL is the primary language that was used by lots of developers and when we had a spark SQL that was available then it becomes very easy to interact and write a customized yeah this Pathways to process the data without going into the writing the customer jobs in 2016 uh in the same Journey we had an Apache flank that was introduced and Apache flank uh focused on having um kind of a Lambda architecture so that we can process not only the stream data but also we can process the batch data so you can see the trend is basically simplification as well as catering to different needs of the big data processing the streaming data as well as as the focus on batch data processing at the same time then in 2017 we had an Apache being that was introduced and it basically uses the sum components on for flank but at the same time we had ETL capabilities that were also available in Apache beam so you can see across this uh timeline uh we were trying to not only optimize the processing of data but also we were focusing on simplifying uh the data processing across this journey so having said this kind of a stage for the introduction of a purchase box let's try to understand what the purchase pack is and why it's important and what are the different components that are available in Apache Spark so Apache spark let's try to understand what the budget spark is Apache spark is a multi-language engine so when I say multi-language uh it supports both um multiple languages for example python R Scala Java so it provides an abstraction uh or kind of apis to interact with the engine in the language that you would prefer to write your code or interact so it's it's a multi-language engine for executing data engineering so when I say data engineering we are aiming to process multiple types of data we want to process batch data we want to process the streaming data we would like to process um unstructured data coming in the form of Json or Json files so there is a huge varieties of data that we would like to process we might want to process graph this data where the data is stored in the form of nodes and edges and in the data science areas and machine learning areas we may want to perform lots of different type of prediction algorithms you may want to come up with some kind of predictions for the loan applications using low stake regressions or we may want to go with the sophisticated algorithms of xgboots or adiables it could be anything so what does Apache spark did it provided a multi-language unified engine so that irrespective of the language in which you are comfortable well with you can use Apache spark for a variety of use cases of data engineering data science on a single loan machine or on the cluster so it's important to understand on a single load machine or on cluster so you could either use a single machine in which your master node and the slave node will reside within the same machine or you can use a clusters and technique thousands of machines which can be acting as a worker node or one of the machine from this cluster can act as a master node so it's important to understand uh here uh what the patches are provided and so we can see a purchase provided they have language choices um data processing choices different use cases and the latest version of the Apache spark that is available at this point is 3.2.3 which was released a couple of months ago and then the first version was at least um 10 years ago which was Apache spark um 0.6.0 now let's try to understand its timeline as well so we can see here that we had a map reduce paper into government 4 which was um came into an existing by uh Jeff and Sanjay and after that the actual spark paper which was came into 2010 by zaharia then in this um spark paper it was conceptualized that what if we don't write the intermediate Research into the persistent storage already and thereby reducing the i o operations and keep the intermediate results within the memory to allow the in-memory Computing for the efficient and the faster applications processing and and then in 2010 the Apache spark was open source and in 12 we had an rdd paper that came into an existence in RDG paper uh it was conceptualized that we we will have a resilient distributed data sets or our data will be partitioned and distributed across the different nodes which should be resilient so that we can compute uh if they are lost because we are working within the memory or a ram so data might get lost it's important to maintain its lineage uh at the same time uh during the next year we had an spark streaming paper came into the picture by zaharia and then you can see pretty much the timeline is same and we had the Kafka that was introduced in 2011. so as the data types were changing use cases were changing on the research on the spark framework but also progressing and then in 2014 it became apache's top level project and after that we can see that for each year there were lots of different um uh application available uh processing capabilities that were being introduced so for example next year we had a spark SQL paper and then we had graphics paper that came into the existence to support Graphics libraries and then we had tensorflow paper which was raised to uh which came into existence into 2016 for the Deep learning use cases and then we had the Deep learning pipelines which came into existence in 2017 to support tensorflow or forward libraries on the Deep learning use applications so after that let's move uh to understand uh the different aspects of Apache spark ecosystem or what the Apache spark ecosystem consists of so you can see in a pretty much in the patches part ecosystem we are focusing on the different aspects of the data processing as well as the management of cluster so starting with the programming layer of the Apaches back ecosystem you can see it's a single unified interface and provides a choice of different languages for the development of your spark jobs you can write your code in JavaScript python R in any of the language you are compatible with and it provided libraries for the different use cases so if you want to interact with um uh spark using SQL you just go ahead and explore using Apache uh spark SQL library and then if you're working on the graph um the systems creating nodes and edges we have a graphics Library available in the purchase Park for streaming use cases Apache streaming is spark streaming is available and for data sense applications we have MLF libraries that can be used for building our data science models uh speaking in terms of engine so Apache spark core is the core engine which is common irrespective of the type of libraries you are using all the programming language you are using your core engine will remain same it won't vary it doesn't depend on the library or programming so underlying institutions happens within the jvm environment which remains same and in order to manage your cluster so if you are working in a cluster mode you are free to use any of the cluster manager so for example you can go with the Hadoop yarn to manage your cluster or if you are working on a standalone mode you can actually focus on uh spark Standalone cluster as well uh speaking in terms of storage because it's important to understand at this particular point that spark is a processing engine it doesn't contains a persistent storage for uh storing the data so you need to have some source as well as sync aware from where you are going to read the data and you will store your processed data and Spark allows us to have an interaction or with multiple sources and things so your data source could be a traditional CSV or a txt file or you might be reading data and writing data to Hadoop hdfs layers or you might also interact with a data storage layers or available in Cloud so for example Amazon has three or Azure blog storage or it would be a nosql database as well for example you might be reading data from mongodb so here you can see in this ecosystem there is a lot of flexibility available at each layer and depending upon the use case upon the sources and the sink that you are using and convertible so after understanding the Apache spark ecosystem offerings let's try to deep down a little bit more on the distributed processing and Spark how this this distributed processing in a spark happens so all the applications Whenever there is any request goes for the data processing it always goes to the driver node or the master node to the system so masternode is the main method uh is the main uh node which consists of the main method which actually performs which starts an execution for this so you can see within the master node we have a driver which consists of spark context uh as a first program which you can think in terms of it's a main program which gets executed whenever you interact and send your query to the driver node and this Spa context or the spark session provides a Gateway for the further communication with the cluster manager why I mentioned cluster spark context or session spark session because before spark 2.4 uh we had a spa context available for this uh as a first session at another session um gateway to manage to have a communication with cluster manager if you are using a code native apis and if you are using your spark SQL you will have the SQL contacts so there were different um spark Quant uh Spa context or as per context available or the functions available for the different use cases so in order to simplify that uh after the spark 2.4.0 uh spark provided a single spark session now you don't have to create your own context for each different use case you can just go and kick off your spark session uh within the diver program to interact uh whether you are using um spark SQL or you are using ice spark or spark are irrespective of that type of context it's the spark session provides a single Gateway for that for the communication with the cluster uh all the communication that happens within the driver a node then it actually demands for the resources uh from the cluster manager so if you are let's say reading the data you or you want to process the data so it needs to get some kind of a worker notes or an executors to process the data so it interacts with cluster manager using your spark sessions and asks how many workers can I get can I is it possible to get these many workers and then the cluster manager manages your resources and it has this information of of how many worker nodes are up and running and it provides this information back to the driver node and allocates those worker nodes to the driver nodes within the worker nodes we have the set of processes which are performing the actual computation and the data processing and these processes are termed as executors and within these executors uh the data is that divided into the smaller smaller chunks and this processing and the computation is also divided in at the lower scale which is known as task So within the worker nodes we have a set of executors and each executor will be performing a certain task in parallel so if in order to summarize the entire processing until now so we had the driver node which receives a request for data processing from the client applications it requires for resources using cluster manager and cluster manager allocates these worker nodes to the driver node for the data processing and provides the information of how many executors are available and what are the different memories that levels that are available and then the driver node allocates the work to these worker nodes worker nodes performs the computations or the data processing in parallel and then sends these results back to the driver node as the physical and the logical plan that was created by a driver node so we'll Deep dive more into this architecture and I will just revisit this in the subsequent slides as well so how this uh physical and The Logical processing happens within the spark environment so you have a user code here in the left um right corner and then just use a code when it goes to the driver node it's a completely unresolved up to that particular point and drive what the driver node does the driver node needs to First validate whether your unresolved logical plan is valid or not so for example you might be reading a file from a location which doesn't exist or you might not have an access to read the data or process the data then it checks that particular metadata information from its catalog and then after its analysis and if it's valid then it goes into the creation of resolved logical plan or how the data is going to be read once on the reserve logical plan has been created then it goes to the logical optimization to create our optimized logical plan and this particular point it's very important to understand how this logical optimization happens because this is the core of data processing in a spark so spark internally uses tungsten and Catalyst Optimizer so what the Catalyst Optimizer does it tries to apply the code optimization and strategies for example constant folding or removing um the parts of code which are not being touched on so various um code optimization strategies are embedded into the Catalyst Optimizer which tries to optimize this resolved logical plan and what the tension Optimizer does it tries to come up with the memory management techniques so for example uh how many how the garbage collection is going to happen how many um how the memory management is going to happen what is a heat displace that is going to be available so all the memory management um optimization happens through the tungsten Optimizer whereas the code optimization this goes through the Catalyst optimizer now up to this particular point you can see we had an optimized logical plan here oh sorry yeah so we had an optimized logical plan that has been created until now uh but there could be multiple uh physical plans that may be available after this or from this optimized logical plan so what the spark does it creates the multiple physical plans of the query execution and then Within These multiple physical plans it calculates the cost of each particular plan um for the data processing and then the cost model decides which physical plan is the final plan to go with and then we have the best risker plan that has been identified and then using this best risky plan driver nodes decides the data processing steps to be executed on the cluster so you can see like here that it in the entire execution of the patches path processing we had certain optimizers that are available and this is um the beauty of using Apache spark and the constant work is being done to improve uh the optimization process of creating different optimizers so in the project hydrogen we are current uh aperture spark is currently working on supporting deep learning uh various use cases because we also want to have a good communication across different um um private across the worker nodes uh in the Deep learning use cases the reason is so for example if you are training your neural network or a reinforcement learning model you may want to use your previous learning to go ahead with the further subsequent processing so it's important not only the distribution of data across the different nodes but also how these worker nodes can communicate with each other and can use the Plus data that has been processed by one of the worker node to be used by another worker node because there is a strong interlinkages that is dependent across the worker nodes I would definitely recommend to check out the project hydrogen um processing on the Sparks website now let's move ahead and focus on this fact terminologies which we encountered earlier so the main terminology which you may encounter in using a spark is rdds because rdd forms the base of all the spark data processing so rdds are the main abstraction of the spark and it's a resilient distributed data set which which is a collection of element that has been partitioned across the nodes so all the data that you read within the spark is ultimately processed and the form of rdds and these rdds are created uh when you read the file and put that into the spark data spark data sets and so rdds you can think in terms of java objects which are being processed within the jvm environment so it's the lowest level of abstraction that is provided by spark and all the processing ultimately happens in the form of rdbs but having a rdd manipulation using the different objects is really difficult because you have to understand about not only um the tuning of their Java objects within the jvm area but also you will have to focus on the native uh the customized Java based you need to know basically about the Java processing so this path came up with the data frames which is a higher level abstraction after 1.6 is it is in line with that we wanted to simplify uh not only the data processing capabilities but also we want to do it at ease so the spark provides different data frames available in Scala Java Python and ours it's pretty much similar to your uh the data frames that you encounter in pandas so similarly you can use the data frames of java and or are it and also it includes some kind of um optimization engine techniques so when the data when you interact with the data frames it your underlying uh data execution is in her inherently gets optimized so it's always advisable to use data frames until unless you have a specific use case where you need to interact with rdds and process the data at the lower abstraction level most of 99 of the use cases developers interact with the data frames and it provides an inherent uh as spark S12 optimized execution then you might encounter with the Transformations and actions so transformation is a function which produces a new rdd from an existing rdds so there is an important concept about rdds rdds are immutable so once you have created an rdd you cannot change it you rather you can create a new rdd using that particular rdd and these Transformations are lazy in nature when I say lazy nature it means um if you have created an rdd it is not going to be executed immediately rather it will just create a plan and just store it itself it will only be executed if uh when you will call an action or if you want to um get the results of your rdd creation and project it on your on the driver node so there is an some advantages of using the lazy execution uh approach while working with RDS the main advantage of having a lazy execution is that it provides a resilience to your system so if since we are working in an in-memory compute in memory a processing and during your memory operation that if there is any failure happens your data gets lost you can think in terms of your working and uh the ram area and the data doesn't processed for a very long time so you need to know how this rdd was created and then it provides a kind of a resilience so whenever there is any loss happens it just recomputes itself and creates the rdds and what are the actions so in the transformation we saw that one rdd gets creates another rdd but when you want to work with actual data set at this particular Point your action is performed and the actions are the spa rdd operations that gives the non-rdt values so for example you if you are counting the data or if you want doing some kind of an aggregation and you want to project that doing at that particular Point all the data from the driver nodes gets uh executed and then it's returned back to the client application so this is known as actions in terms of Spark so we you might also encounter on the job stages and tasks because this might become sometimes very confusing over the jobs and stages and tasks are um jobs are uh whenever you perform any action uh the spark job is created so jobs is nothing but uh the kind of um the package of work that submitted to the spark and these jobs are created at the driver node by the driver notes and then each work uh within the job is then broken down into multiple stages so how do we decide uh how many stages we should have within a particular job it is based on the shuffle boundary so if your data needs shuffling across um the different nodes then all that data all that work is tagged into the singular stage So within the jobs you can have multiple stages on the basis of the shuffle boundaries and then within each stage you can have uh some tasks that are relevant to a particular uh processing area so we can consider this task as a smallest unit of work for a spark and a single attach the single operation that is applied to a single partition or the single area of data we have a very um small diagram here we can see that we are the driver node we have got multiple jobs that has been submitted to the travel node each job May consist of multiple or singular stage and within this stage we can have single or multiple tasks that are available and here at the bottom right you can see like we have a driver program which is interacting with the cluster manager using the spark context and the cluster manager assigns the resources or the work on which the driver program for the actual work processing and within the worker node we can see like we have a set of executors or the processes that are available for the actual work and do for each executor is associated with a cache memory which is capable of handling some of the tasks so and each worker node performs the work that has been assigned by the driver node and sends us it's routed back to the tracker node so this is a kind of summary of entire spark internal processing that happens uh when we submit any of the spark jobs to the driver program let's move on so until now we saw that how the spark um processing happens let's try to think around and understand how can we improve the performance of our jobs you might think around um okay so if there is a parallel processing that is happening across the multiple uh more note why we should not um have any kind of interdependencies across the different worker nodes so which means you would like to reduce your expensive Shuffle operations for example worker node one is doing some work which is required by worker 2 to begin with in this manner you will not be able to achieve your parallelism or this is in terms of in terminologies of spark we may want to reduce the shuffling of data or data redistribution every time we cannot always avoid that if we have these dependencies uh for example if you are using operations like group by key already used by operations or joining across the two data sets where the data lies on the different nodes then you Maya you will have to do that but in certain areas you may want to avoid that and this is something which can be configured within the spark using the shuffle partitions configurations so we need to optimally tune our property of Shuffle partitions to improve its performance um another thing you may want to think around why to read the data from the hard disk every time why not cache it if we have a data frame or you can see here we have a data frame why to read the data from raw data again and again every time to create the same copy of the data yes you're right we can uh read the data once we create our base data frame and then we can just use put that data frame into the cache memory of the executors that we saw earlier so every time we will need uh the copy of this raw data frame we'll just go ahead and read the cache memory read it from the cache memory which as always um faster so spark provides the multiple storage levels um so for example either you can read the data from memory only or memory and disk if the data cannot fit into the certain memory area and then we get at the same time we have six uh seven uh storage levels you can see here uh which can be used uh for the spark Performance Tuning let's look at the next um tuning aspects which you can use to see live data format and this is a very common um while working with a spark or data processing uh we should always read the data in a kind of serialized data format so when I say serialized data format it means in the data format which is readily usable for the next pipeline stage so consider the data pipelines in such a manner that you are processing through some data which is going to be used by another part of your pipeline so if you are persisting your data in such a manner that it is readily available and there is no operation involved of serialization and deserialization um you will fasten up your spark processing and it will improve your um processing capabilities so for example you may want to go ahead with the optimized version of data processing frame formats having a packet is a common uh method of serializing the data and average another so it always depends upon the use case which format to go with but most likely if you implement a serialized data format then it will improve your processing of data and then we have we don't want to log out log everything that is happening within this spark processing so you can disable your debug information login and just log out whenever there is any importing happens and it does saves lots of resources because every time you enable your debug and for logging it goes and saves into your hard disk or storage which is an extra cost for the system so again IU operations is something which we can optimize at this point you may also want to use the proper partitioning strategy so for example using repartition or collect coalition uh so in the repartitioning there's a cephalo shuffle operation that happens so here at the top right you can see here we have uh four data sets um which are partitioned on the different um uh values and if you want to reduce the number of partitions always go ahead with the callus operation because in the coordinates operation we are going to have the union of the data sets rather than if you go with the repartitioning then everything is going to have a data reshuffle and after the reshuffling then the partitioning is going to happen so you may want to avoid the unnecessary full cluster scan so other important thing which is I briefly touched upon earlier as well is to always go with a higher abstraction levels of implementing your spark data processing so for example you use data frames or data sets instead of doing uh your data processing using rdds because rdd is um are not that optimized as compared with the data frames and the data sets and also if you are reading data from some of the database which is very common uh in terms of the spark processing you may want to use your push down optimizations to extract the capabilities of your underlying databases so for example if you are using Tire data you may want to use some of the capabilities which each database offers at the same time you may also want to use the partitioning and the fat size um uh hyper parameters which you can tune out to improve your performance of your leads and rights from the spark this is another important point you uh you should avoid the user defined functions because user-defined functions is something which causes um data serialization and these serializations every time so every time you use any kind of user defined function it's really expensive and the real cost lies in the serialization and the tclage so it's always better to go with spark provided functions to improve the data processing then we have a China strategies which is another important point you might want to think around so if you have two big tables and joining together then there's a shuffle join that is bound to happen but you can tweak out your parameters up to which particular point the shuffle joint should happen uh if there is a big to a small table join then the broadcast join allows you to copy the small tables over each worker node and then it will prevent your unnecessarily data shuffling and will improve your performance so let's have some quick uh exercises on the big data processing using collab I have got some data which has been downloaded uh from kaggle and we are going to download some data from kaggle and then we will do a quick demonstration on the data processing using cadoo so here I'm just installing um kaggle API and then after that we are going to have in order to download the data from Kegel you will need a kind of an authentication or a credential file every time uh so I have already downloaded this uh into my session so let's pick that so I have the Kegel so here's a credential file which will be unique um for your system and you may want to just write that so here we will create a kegral directory into our system and then I just change the permission but since I have already created it will not allow me to do that but that's okay here we are going to download the data from kaggle uh which is uh daily temperature data sets of all the major cities and countries and it's available as an open source uh and free to use you may want to just go ahead so since I had already downloaded it and I unzipped this into my workspace here so you can see I have a temperature City temperature CSV which I have got here and in order to work with um spark we need um jdk that is suppose that is need to be available in our environment and then we have the we need to have the pi spark that is available it's very simple to install it's just a python package by spark you can install and mentioning the version so I'm using here the 3.2.1 which is a stable version of by spark and in order to check if I have installed it correctly I can't run this Pi spark version to identify ours uh what's the version and if it has been installed successfully here as I mentioned earlier uh python is has provided us a spark session as a gateway to interact with um the cluster manager so I am using the spark session uh we are currently instead of a smart contact that's another thing to watch out and here we are just reading the data from the city temperature csvj file that I have just downloaded from table let's try to inspect the schema using the print schema function we can see that it contains region information country State and the average temperature until now for each month and here if we try to run uh see the five top five rows of our file we can see that the format is a row format so now since we are interacting with the data frame which is a higher level abstraction so it um it stores the data in the form of a root Java row data object format and if we use the show function then we can see that we have got some the top five rules of this particular data set and if we uh want to use the pi spark here uh that means we want to perform the data processing using python so I am using the pi spark Library here and the SQL or types of integer and the float type you can see here that this particular average temperature is a string and I cannot perform any kind of an averaging operation or a function on the numeric column on this string type column so I will have be converting here to the flow type which I just did and then let's see how it looks like so we can see here it sent converted into a float type column so during but here we can see that it's kind of uh the temperature is given in the fahrenheit and we need to convert the data into scale so just use that um simple logic and formula to convert it into uh Celsius scale and see we can see here it like the average temperature converted is now in Celsius degrees Celsius and here if we want to let's say um explore this data using the python API or Pi python API as I said earlier it's a unified engine uh you can use any of the language to interact with so let's say I'm focusing I want to use pythons I can simply interact with my data frame by data frame using python by applying um the filter for all the region for all the countries within the Asia region and then I am grouping here on the basis of countries and here and I want to find the aggregated average temperature for all the countries uh within Asia region and here it's an important point now here we have got um the average temperature for each country for each year from the Asia region but I also want to understand how it has been calculated internally so here we have a DOT explain function which you can use in spark to come up with an execution plan that it uses underlying so we can see here it initially performs the file scans then we filter out the temperatures um where the app and casted the temperature as float and then we also put the filter on the region as Asia and after that it projects the result set containing these columns only um and then we perform an aggregation on on the basis of partial uh on the on the base at the level of country and year and the functions so it has a partial average function so you don't have to calculate the actual exact average of your temperature rather you can focus on um having the quick partial averages that is available and it's approximately the same uh of your average function and then there is a hash partitioning exchange operation that happens after this we can see here then the final aggregation happens and here we apply the average um temperature conversion uh that is ultimately being uh because this is the data data set that has been resulted and returned to the driver node from each worker nodes and once the computation happens at the driver node we can see here then the X after the range partitioning it also performs a sorting operation because we have mentioned it to be ordered by to the country at a country and year so it performs the Sorting operation here and then it results sends you a final results so it's important to understand how it is processing internally uh when we process this data but until now let's say I'm not proficient with python I am comfortable with SQL so as I said earlier spark also provides the capabilities to interact with your data using SQL so you may want to just first you can just persist it in a kind of a view uh and then we can give uh the new name for this view as temperature and we can interact and play around with the data using the spark SQL API so for example here you can see like I interacted with um for the with the data using the simple asteroid query where I aggregated the countries uh and the ER level and calculated the average temperature in degrees Celsius for Asian region but we also need to think around uh do we have the same explain plan or the process does the internal processing happens in the similar manner we can compare here the execution plan for this and it came out to be the exactly same how we got it from the python based processing so we have the exact same steps of file scanning filtering projection and then the exchanging of partitions and after that sorting operations so here we can use the same spark SQL to uh to come to have any kind of a queries after that we may also want to think around what does the spark allow us to have the complex as well queries because you might be using CTS sub queries it just allows that and it's just that the explain plan becomes a little bit complex uh depending upon the query that you may you are executing so for example here let's say I want to identify for all the Asian countries uh uh and want to perform an aggregation of year and countries and corresponding average temperature whose average temperature is higher than the global temperature so if your valuables with a CT you can figure out I'm finding it here uh an average global temperature and then picking all the countries whose uh average temperature is higher than the global average temperature and then sorting the order or sorting the results on the basis of the average temperature and if I run this query then we can see that um the top countries and the year that they had the highest temperature until now so we can see it's Thailand uh and Indonesia they are the countries which had a higher temperature until now you can use uh the describe operations like we generally do in databases to configure the schema of the objects or the view that we have created and now let's we can also uh explore some of the internal processing uh though we don't have lots of time left but let's quickly go through that in order to interact with your respect your UI you will need um the find spark application or package that is provided for the inter interaction so here I am importing the pi spark package and after that you have to download and install NG rock this is the step that you will have to do if you are interacting uh within Google collab but if you are have set it up on your local system then you won't have to do the system so once I perform this operation and I will be in the end I will be getting a public URL for my spark UI so if I go back if I click on that um uh spark UI URL s again let's try to see okay so here so here you can see all the spark jobs that has been executed until now we can see there were 28 jobs that has been executed within the shell and if we just pick uh the first job which was a very simple job just to read the data from the CSV and put that into python data frame it we can see uh the stage what are the different steps in which that were involved in the a stage execution you can see here we had a file scanning operations and then the map partitioning operations uh that would happen so what we did we did read the file we partitioned it across a different uh weapon nodes and uh this is uh the stage that was responsible for this operation we have some of the matrices which are available for each particular stages and at the executor level information uh though we are running out of time we can quickly look at the different options that we have uh in the spark UI you can see here we have multiple memory level tweaks uh that you can see here that has been done and at the same time we can also get executor levels informations that has been used so that's a quick one um to explore on spark UI all right let's go back to our presentation okay so after this exercise I will just want to leave you with the one single thought that smart is a processing engine and doesn't uh contains a coupling of uh permanent storage and the compute like we have other resources that are available till now and all all of your code executes on the jvm so whether you use python or R you will be happy using internally the jvm to process your data and it will be a spark session that is going to be created as a gateway to have a communication with the executors uh I all the diagrams which I used were not mine so I am thankful to the various people who created those diagrams and I refer to these um different resources for the preparation of my slides yeah thank you so that's all we have we can take some questions if they are before we proceed to answer your question uh I would like to request the written dish to please fill in the poll about feedback as it helps us to conduct more success foreign okay the first question is the fundamentals and then the framework so I did covered some of the fundamentals uh but it was important uh to give you an overview of the framework and the first um stage so I Believe by now you might have a good understanding of the fundamentals as well as the framework uh let's move on the second question what do you mean by ecosystem so when I say ecosystem uh ecosystem contains all the different components it doesn't means only how the processing is happening uh we also focuses on the different um parts of uh entire uh area like for example if you are using your cluster how are you going to manage your cluster if you are using your processing then we have a spark core which is responsible for processing if you are interacting with a spark in a particular language then we need to have an API to interact with that language so all this constitutes and is path equal system why being was introduced so beam was introduced as part of the different requirements that we had uh earlier so for example you may wanted to have stateful computations we may want to have the streaming data at the same time batch data processings at the same time you might need the ETL processing to be embedded for those use cases and scenarios so if spark B is actually um the kind of new technology that has taken care of lots of different areas as we have evolved across the data pipeline Evolution Journey is uh this battery beam provides the same functionality as Apache spark yes it does uh since uh you can use um the streaming capabilities as well as um batch processings at the same time you can use uh ETL uh processing so it does provide the same kind of functionality but you have it is aimed to serve for the different purposes because you may want to focus on the patch processing streaming process within the same uh area rather than having the spark streaming or the spark core uh every different altogether is so into in short yes it does provides uh the same functionality as this path uh what is the difference between partition and parallelism so when I say partition it means your data is going to be divided and will be put into the different chunks into different uh parts of the worker nodes but when I say parallelism so parallelism is uh usually focused on how you're processing that is going to happen internally so another concepts of parallelism is that you may want to distribute your workload across on the different parts of your cluster uniformly so that's where the concepts of parallelism comes into the picture and partition is one of the way uh to achieve that parallelism okay uh let's pick the next one how to upload data set from desktop to collab you can use um from if you are interacting with collab then you can directly upload it within the runtime environment so just click on upload button at the top and uh it's very easy to upload your data set from desktop to cooler but if you are uh have some kind of a data which you want to read in the form of API s