Cloud Dataproc - Migrate and Optimize Spark Workloads | DataHour by Ritka Neema

Key Takeaways

stable and efficient so with that let me share my screen just give me a moment cool thank you thank you I hope I'm audible now okay can I get the confirmation in chat if I'm Audible foreign okay sorry for the delay okay um so as you're aware about the topic that we have for today let's go through the agenda uh we'll quickly go through spark what advantages it's provide uh what advantages it provides and what are the available Integrations it would be really interesting to see how many component it allows you to connect with then we'll see a data proc for Hadoop ecosystem why there is a need and what are the advantages there uh we'll see how you can migrate your application what are the steps you can take to optimize it and then we'll see a quick demo which would not be running an application but uh just to familiarize you guys with data Pro Qi and how you can utilize that to uh learn more about it with that uh let's start um so Apache spark is well known as a in-memory competition engine it is the most popular framework right now for processing your data companies like Google Netflix Twitter all of these companies are using it be it a small company or a big company if you see the opportunities job opportunities for data Engineers Apache spark is a key requirement there so um when I say a purchase Park is an in-memory computation engine let's understand uh first why there was a need for this traditionally mapreduce was used as a part of Hadoop ecosystem which involved computation on the Clusters whereas the data lied in the sdfs side so whenever you want to process some data you read it from the i o uh and if there are any intermediary results which we usually called as Shuffle data you write it back to the i o what essentially happened here was there were a lot of IO read and writes and these are expensive operations in terms of time and cost so uh this is one of the thing that Apache spark addresses now back to the definition in memory Computing engine or built-in memory processing engine what aperture spark does is you have a cluster you have some data nodes and Within These data nodes you have RAM on top of on which your computation will happen so say your data is lying in htfs or GCS your application have read the data from that particular location ones that have read the data all the intermediary results are only written into RAM there are certain configurable properties so that if you want to write it to hdfs in intermediary results or to GCS or to local ssds those are possible but just to understand that the main computation that were happening before that involved external i o those operations are not happening here so as a result a purchase spark is 100x compared a faster compared to Apache Hadoop so let's move on with it and see what it provides us uh it essentially Apache core engine layer is there this layer is the one that basically provides you in memory computation uh this is the layer that provides you I O scheduling monitoring logging so again then this is the main layer that gives you um gives spark most of its properties right then on top of this we have spark SQL so um when um in a data environment or in any company if you have a lot of data there would be certain business units or certain uh stakeholders that they would want to query the data and see the results a lot of the times these these go to people in form of dashboards or a final table but sometimes uh especially data Engineers or developers who are through thorough with the process they do want to do interactive queries even business users want to do interactive queries with your data so this is a SQL shell on top of um spark core that allows you to directly query your data and it gives you a syntax like SQL so anyone who knows SQL can easily use spark SQL okay then uh we have spark streaming uh sorry then we have spark streaming which basically uh which basically allows you speed up to gigabytes per second what else you want you want a technology that lets you do batch processing and at the same time streaming and interesting thing is there are not a lot of changes when you're trying to do batch processing versus spark streaming uh the interface or the API can remain the same there is a different API as well but in most cases you can use the same API so uh the same way in which you are using batch processing with a little more functionality learning a bit more about what are the functions and understanding of spark scraping streaming you can just start straight away start using spark streaming then you have spark ml lip it would be interesting to know that Netflix uses its recommendation engine using this particular Library um then we have Graphics which is basically for graph Computing or graph networking any questions still here in the chat okay let's move on so now you've understand what all things spark provides you uh what it is what advantage it have over my produce let's see what are the different uh what are the different Integrations that can provide you so with a single technology or single processing engine you can connect to a multiple sources you can traditionally hdfs Hadoop or Hive is something that we have been using Apache spark for but if you want to use it for streaming data coming in from sources like Kafka Google Pub sub or Flume that's also possible any SQL database that is any jdbc compatible database is also something that is supported so uh be its SQL Server MySQL Oracle or any other uh or any other jdbc database you can simply manipulate that data process that data and use it using Apache spark um then a lot of data warehouses um traditionally we will be using it for hype but even things like bigquery uh it's possible even nosql databases like Cassandra even no SQL databases like Cassandra are supported and um like most of the companies or people are using Apache spark it's suppose object stores so for example in Amazon it is S3 usually people keep the data files in S3 like in Google Cloud people you keep their data in cloud storage bucket um so again this is a very interesting thing because with one processing engine and just with the different set of connectors and just changing the class that you're trying to connect with uh you can connect with any source uh now that we've learned about um advantages of spark and the Integrations why would anyone run something which can run on on-premises on cloud or say for data proc uh so let's try to understand the need of a data proc right sorry so for the people who have been using uh spark on the on-premises or Hadoop ecosystem on the on-premises they know the pain associated with it right we've all been through the pain of scaling it pain of having it set up so initially when we are setting up a Hadoop ecosystem there are a lot of challenges you first of all you need to have the physical infrastructure at your data center then you need to manage all the security you need to manage all the um say version compatibility all the jars all those things you have to manage yourself uh this is on this is not completely utilizing the capacity of the developers who are trying to uh process the data for some business decision or for some final work uh a lot of the time uh we are using efforts on infrastructure thing which is not the point uh second uh the total cost of ownership you have to give a upfront cost because you have to have the infrastructure at your place um that is an added cost for you right with Cloud um you can pay as you go or there are multiple other options that can make this a much cost effective solution and uh largely scalability has been an issue with on-premises system so when you want to add some notes first of all you have to go through a long process right you have to get the approvals but even you've got the approvals to physically add a node and then you have to use it what happens between the time that you need a node and it is actually available for you to use what happens to your data or the slas so reaching slas or completing the jobs on time have been an issue on the on-premises system foreign from Google Cloud which is basically a managed Hadoop solution all you got to do is uh run the Clusters with a few clicks or if you are comfortable using CLI just a few commands within the CLI and you have a infrastructure that is being managed for you but that will run for you you can submit any job you want or you can have the components that you want while creating the Clusters so this is one of the things that we'll see at the end of the session today now uh let's see more about data block or exact advantages that it offers to you first of all it uh is open in the sense that it allows you to integrate open source technology or open source data analytics uh very easily uh flexible I'd like to take it at last uh then it's intelligent you can use it with components like vertex Ai bigquery and data play uh like I mentioned before Integrations are really easy it is mostly like Plug and Play or uh just few config settings and you're good to go so with bigquery um again it is kind of a where uh warehousing system with added power of ml uh so when we see people who are migrating from uh on-premises systems to Google Cloud they might also like to replace their hive warehouses with bigquery because of the analytics Advantage cost advantage and the ml advantage that it provides then for any other specialized ml use case you can use vertex AI which is again a very easy to use so that makes um the environment and that makes data proc a lot more intelligent and for the people who have been using Hadoop on the on-premise systems we know how we got uh X is there right uh tokens SSH Keys you have Kerberos authentication all those things and in critical environment when we in Big Data we are already handling critical data we do want to maintain that security so that's again the advantage or the basic uh thing uh with additional security features by dataproc uh it's there and then we have cost Effectiveness like I mentioned total cost of ownership in on premises systems is really high um with something like data proc you can bring it to as low as 54 percentage there are multiple things that you can do for this one of it is commitment discounts so um say if you know how much uh compute resources you're going to use you can kind of commit that hey I will use this much and accordingly um you can get a discount for that or even if not that we have per second pricing that allows you to make it cost effective now lastly when we say it's flexible now we know it's a managed solution right but uh sometimes you might want more control over it or less control over it so we have that option as of like as flexible as a developer you can choose to create or run your clusters on Google compute which is Google compute engine or on Google kubernetes or serverless now the first two options are something that we would already know about um anyone who is already aware about serverless here in the chat foreign so spark serverless is a feature offered by data proc that basically makes your spark workload a serverless now contrary to um for running spark workloads of course we do need server right but the term serverless uh comes from the fact that you don't have to manage the infrastructure a lot of the time that you're running your spark workloads you are left um fine-tuning your job or just right sizing your clusters so that headache is not there anymore if you know you want to run a job you can just submit it using the spark serverless thing uh using this particular feature and that basically makes all that pain go away you just have to submit your job and it would be handled uh all the infrastructure would be handled including Auto scaling again if you want more control in serverless also there are some features um of course you can give which region you want even if you want to specify which machine type you want any additional spark properties those can be added so this makes it a lot easy for the developers as well as a lot configurable now if you see here on the right side there are a lot of things that dataproc can work with all these components can be enabled and this is again something that we will see at the end let's talk about application migration uh when you are migrating on premises application to Cloud to data block there would be a lot of considerations you will have to make a lot of these considerations can be uh which component you want how whether you want to lift and shift whether you want to redesign or modernize your application uh this Slide by Priyanka who's also known as the cloud girl.def she beautifully explained what would be few of the considerations and in which cases you would have to use data proc um and to summarize in just one sentence uh lift and shift is the absolute case where you don't need to think much and you know you can just get your whole application and get it to migrate to uh data block without making major changes in some other cases where you're redesigning you'll have to reconsider this thing uh like for instance Edge base if you need to uh integrate it with things like Apache Phoenix so you need to use Crow processors you'll use dataproc if not maybe you can go for a managed solution so why not go for cloud bigtable similarly for processing uh streaming data like I mentioned spark um support streaming processing so if you are using spark or Kafka that is also something that's supported by dataproc so any Hadoop ecosystem component any Hadoop compatible thing can be run on data proc very easily which gives you the benefit of cloud and also uh it requires minimal efforts so you can get benefits from a very small time and then we have other considerations like if you want to do interactive querying all right how would you do that you can basically enable a data proc cluster and using uh optional web notebooks which is Jupiter you can just do interactive querying uh to mention spark have been used a lot for ETL or LT processing and similarly it can be used in data proc for the same the most common architecture that we have for a purchase Park on data proc uh is using Google Cloud Storage instead of hdfs now uh how that would work if I'm to show you the next thing that would be for migration is migrating the code if you have a job on premise you would be referring to the data in is in hdfs using this syntax sgtf is colon colon you just have to get those files to Google cloud storage and then you can just start referring them as GS colon slash so this is the minimal change that you have to do of course when you are migrating data at scale there are certain configurations you want to automate these things you want to have some mechanism for copying the file from on-premises to gcp uh usually it is this CP but there are many other um tools or components which also support the same job again to talk about migration and the most common architecture uh people do use um clusters on which are ephemeral clusters we'll talk about that later they'll use GCS instead of stfs and for meta store instead of um using using the one that is directly within the data proc they use something called Data proc metastore service so that's about migration the common practice or the considerations any questions here okay then uh let's move on we do have to see once we are migrated we also want to optimize so that our final cost in the cloud are equal to or lesser than what we had on the on-premises uh application optimization will talk about it in two different ways one is since we're talking about Apache spark workloads we'll just see a quick uh few pointers on how we can optimize our spark code or clusters and then we'll see what are the features that are provided by dataproc that we can use to optimize our applications on data block okay so let's see how we can optimize this barcode um as a best practice when you're developing spark applications your optimization should start from there so whenever using when you are writing code you should use some best practices like um a lot of times we are probably reading a table doing some Transformations or doing a different set of transformation on that table and generating different outputs so in such cases where data sets are having repeated use you might want to use cache or persist operations again um cache is the one that would have your data uh persistent in memory which is on RAM and persist option you can have different options whether you want to keep the data serialized or deserialized whether you want to keep the data on memory and disk or just disk or combination so all these combinations um are available as option when you're trying to purchase your data based on your application usage you can use cache or persist and persist also with different options okay I have a question in chat which is can we connect external data to from uh data flock to fetch data uh yes you can connect external systems so say you were trying to connect to a hive system on premises you just need to do some firewall settings so that your data proc we clusters can connect uh to The Hive system on premises uh but it can connect uh networking is something that you would have to take care of you would have to think about get approvals for but then we can directly connect with external system to fetch the data you can whitelist this particular IP that your data proc cluster might be having so just a few settings and then you're good to go okay uh coming back to spark optimization um most common optimization technique that's being suggested if you go to any blog or any site or any person uh you have a lot of bikey operations when you're using spark applications naturally you want to uh Reduce by something because the computations you'll have some aggregations and by key operations the most common so you should always use Reduce by key instead of group by key so if we understand the architecture of spark we have different data nodes uh or different cores which are having different tasks each core will have a task and each task would be a processing some partition and if I say Group by key then data from all the different nodes would go to a single node which means if I have four different nodes all uh the data from all my three nodes will go to my one node which is a lot of Shuffle data however when I talk about reduced by key if I have four nodes I would in me I would first reduce in the in that node itself and then get the final results to the one cluster so however in both of the cases you get a data from uh the three nodes to the single node but in group by key there is a lot of data that you try to shuffle so Reduce by key follow the operation first you reduce on your own node and then you send it for Shuffle so that reduces the load of Shuffle and as a data engineer as a as a spark developer you always want to minimize your Shuffle data and then uh if you're dealing with file formats right um like we say for Hive Avro is the format which is most optimized similarly for spark we say park is the most optimized format because it is native to spark so of course it offers some advantages there and then parquet is a columnar format so any operation like filters again which are uh use a lot makes it a lot easy or efficient to do those operations uh then joins are the most joints are also used a lot right when you're joining tables um it might happen that using a large table with some lookup table so if your table is small say 1 MB 10 MB and you can configure this size but don't over configure otherwise your application will explode uh so you can use broadcast join in such cases where you have a small table and a large table to compare and as a rule always have the bigger table on the left so df1 dot uh join df2 and df2 being the smaller one so such things we should always take care of when we are writing the code and then uh I'm not sure how many of you have used spark a lot but if you have used it with a lot of data you would observe for example if there are 200 tasks or 200 tasks in a stage 199 are completed but one task always keeps running this is a signal for data skew which is nothing but uh it indicates that a lot of data exist in a single partition compared to other partitions so naturally for a lot of data a lot of time would be required in such cases you can either give skew hint if it's a joint and if not if you're doing some other operations and the skew is degrading your application performance you should use something like isolation techniques or salting techniques um next in spark um what we are trying to take advantage of is parallelism so it's very necessary that that you're using the right parameters that you are having the correct parallelism and the level of parallelism is always based on your code and your data so it is uh customized to each application but as a rule of thumb if needed try to coalesce rather than repartition this follows the same concept that we uh um went through in reduced by key and group by key operations so that is one thing next um if you are having a lot of data and you know that partitioning uh buys some columns so say if you're doing reduced by key and you're doing it a lot of times why not Partition by first and then do it it will simplify your code it will simplify the uh execution and it will really make it work faster comparatively but when you are using Partition by look out for skew because that is a high possibility it will be there so When developing the application make sure you are going through that and finally uh other than the default serializer we can also use Cairo serialization which is like 10x faster so this one uh serializes the data in binary format it might not be compatible for all data structures but usually it works and this one is quite a performant as well so these are few of the things that you can do or follow at code level okay I have a question from chat uh what is the meaning of broadcast here uh sorry I uh just skipped over that so broadcast join is basically uh firstly applicability is when you have a small table or a small subset of data that you want to join it with a large table or just a small table that you want to join what happens here in a normal join um large tables uh similar you'd be joining based on some keys right uh and when you're trying to have a matching case you have a lot of Shuffle that's why we say joins are heavy but in case of broadcast join you can broadcast like how you have the broadcast functions in your chat applications you can basically send the data set data set to each of the data nodes so that each data node have a broadcasted copy of that data set uh that node one do not need to go and read data from node 5. Node 1 have the broadcasted copy that it can read and it largely reduces the runtime of application so look out for such optimizations if it's possible do that I hope that answer or your question hello hello ritika are you there foreign attendees or you can do by yourself there are some question q a section and chat also I have answered I think most of it there are few pending ones that I can take right now there's a bit of content that's left do you want me to complete that or we are tight on time no no you can continue uh so let me take the rest of the content and then we can take the remaining Q A's yeah sure okay okay coming back to optimization right once you are thorough that your code is optimized you also have to see how you're utilizing the cluster because traditionally on the on-premises there is a single huge cluster that would be used by multiple business unit multiple applications or multiple double spark jobs so how you are using that cluster is a lot important not just on premises even on cloud or in general how you are utilizing the cluster because it says a lot about the cost and time resources um so always make sure you're not under utilizing the application or over utilizing the application and it can be done by tuning spark properties like executor post executor memory executor memory overhead there's some calculations that need to be done but more importantly you need to understand the application you need to understand the data that you are using how much can that data scale how much can the data be less when you understand such things it helps you configure the property which is known as Dynamic allocation so Dynamic location basically uh takes care of the task uh if task on executors so if I have less data today is there a possibility that I can do the work on less number of executors if I have more data tomorrow would I have the capacity or the enablement to run the same run more data on more number of executors so Dynamic allocation you have properties like Min executors and Max executors um which you can set up according to your understanding of the application that's why uh in data engineering it's not just needed that you write the code and it's done you need to understand the data you need to understand the use and the logics now for course uh you can go for tiny executors or fat executors which is nothing but uh many small executors or a very big fat executor uh it might these two approaches might satisfy your application in some conditions but mostly it will not so the recommended or the mediator approach here intermediary approach here is having five cores per executed of course this number can be reduced based on your application parallelism that means if at a time uh less tasks are running comparatively to availability of executors then you can reduce this number then as a developer you don't want to see anything read in your application but if you go to active spark URL and you see uh red for GCE collection it is a bad thing so these are some properties that you can set so that your garbage collection would be um better if you are having a clusters or nodes whose Ram size is less than 32 GB you can they use this particular uh flag when you're submitting the application what it does is for uh data or for all the pointers it uses four bytes rather than eight bytes so obviously it would save you storage there now a best practice as a spark developer would always be to look at active spark URL see for things there like skew see for things like um executor utilization if you want you can also see the diag from there so um Every Spark developer should go to spark UI to understand their application and to see how it's performing and at a large level at a production level or scale if you want to do the same thing of monitoring you can use profiling tools like sparklens sparkling or Dr elephant foreign features uh which help you optimize uh like I mentioned here uh spark traditionally um or Hadoop applications traditionally were run on a single cluster which was a huge of course that cluster would have multiple nodes but that would be one cluster on which your multiple jobs would be running but and that cluster is persistent that means day and night like 24 7 all through the year that cluster runs so of course there's that physical infrastructure Cost Plus electricity cost all those things are added in that compared to data frock or um you should use ephemeral clusters what are these ephemeral clusters these are only a live that is clusters are alive as long as your job so when you submit your job it triggers creation of the cluster and as soon as your job is completed that cluster dies thereof an important thing to note here is if the cluster die the associated spark URL would also uh go off if there are any interactive like you cannot do interactive queries on top of a femoral cluster these are for jobs so these things should be kept in mind but with FML clusters it really brings your cost a lot down so each application can run at its own time like in on-premises system where your application might be waiting because of a priority application those clashes will not happen of course some quotas might apply but then ephemeral clusters as a whole give you that cost benefit uh next uh you can have commitment discount so um if a company knows that it will use x number of uh it needs uh X number of compute engines or X number of resources it can uh have a commitment discount again that it can commit hey I will use this much and accordingly there would be some discount that would be available and these commitment discounts helps a lot for bigger company so if you are working on in any of those this is something that you can suggest or that this is something that can that the companies can utilize now uh preemptable workers uh this is something that would like to take in detail so what are preemptable workers um these are your machine notes that are that provides you the same compute as your standard notes but as the much cheaper price but via this name differently because uh there's a catch if it is having lower cost and having the same compute the catch here is that um that particular note can be called off by data proc within 24 hours so say my application is running for four hours and it's using preemptable workers the application might run fine but if I have an application which might run like 30 hours there are high chances that the workers would be called off by data proc and my application might fail so this is just the concept of preemptable workers how we can use it we'll just see it in the later section foreign policies like uh like on-premises system when you have Dynamic policy Dynamic allocations which means you can use less number of executors or more number of executors how would that thing apply to data block you're already creating a cluster with X memory and X number of nodes what if I need y number of nodes more or what if I need uh four less nodes it can be configured using Auto scaling policies so basically uh Auto scaling your clusters if you're any if your cluster need to scale down if your cluster need to scale up based on the application requirement this Auto scaling policy is something that we'll see uh in detail with preemptible worker in the next section and last if you need more uh storage you can use uh local ssds you can that is you can attach more storage with your clusters again that is something you will only know once you understand the data but this is an option that you can take leverage of if you have more memory requirement okay and now finally Auto scaling clusters uh what are these I've already mentioned these are clusters that can scale according to the needs of your workloads these are like fire and forget jobs so once you start them or when you once you submit them you don't really need to worry about them because you know that will complete because you will always have the sufficient resources if it needs less it will be less if it needs more it will be more so um this is what a yaml five for auto scaling policy looks like let's understand these parameters the most important parameters to understand are the worker configs um just worker config is the indication of your primary workers so when you're talking about Auto scaling or when you're using pmtable workers there are two type of workers that you'll have one is standard workers which is nothing but your standard uh which are nothing but your standard machines these are primary workers and the second would be secondary workers secondary workers are preemptable machines or preemptable worker nodes these are something that might be called off by data proc after 24 hours but you can definitely take use of when you know your application would run within that time or you can still manage your application using that um for auto scaling you have Min instances and Max instances so for primary how many do you want for secondary water again there would be ministers and Max instances how many do you want so in this case it will go from 1 to 50. I can have minimum one secondly work a note and sorry minimum zero secondary node and maximum 50 and it will be scaled according to yarn memory usage um now if you go and read about this basic algorithm it largely defines how exactly your data block cluster would scale basically um it helps data block understand factors like uh should I how much time should I wait before uh triggering or to scaling should I scale fastly or should I wait before scaling uh then if I have to turn off some clusters or D allocate some clusters should I wait for some time or should I not so all these factors would be understood by basic algorithm next is you have to choose balance between standard and preemptable purpose uh Google recommends at least 50 to 50 ratio which means at least 50 primary workers should be there now when you have SLI applications it might be that you want to complete it under a certain number of time when you're using when you're using Auto scaling uh your application execution time is slightly bound to increase so you should always keep that in mind and any percentage of primary workers below 50 might work but would not be stable and trustworthy for your application foreign workers like 75 25 80 20 according to your application and SLA okay um let's go to the demo then foreign just give me a moment here 's me so I just want to show you guys how the creation of Auto scaling policy will look like you can do it using the UI for ease and I think a lot of us prefer you using CLI we will do it via CLI I've already created a policy here thank you okay I would have to reconnect but this is what the policy looks like you have some worker config a minimum and maximum then you have secondary worker config you would have to provide an ID just give me more [Music] foreign foreign so first of all from wherever you are running the cloud CLI command you should create this yaml file which is nothing but your um contains the attributes of your auto scaling policy these are all the attributes that we just saw in that slide once you have it you can create it uh foreign Auto scaling policies you are importing uh into this is the name of what you are importing your auto scaling policy into you're giving the source which is like which yaml file do you need um gcloud reader to read it into in the region in which your policy will be created and then your auto scaling policy is created and finally you have this name this name you should definitely save it because it would be used if you're using CLI further then this name would be useful if not then uh then still it's always good to have the name right foreign for the demo we are not really going through summation of the actual spark job moreover just seeing what other features that we can leverage from cloud data block uh like we talked about serverless uh this is the spark serverless platform where uh or UI from which we can create spark bad jobs so uh yeah to node so spark serverless only works for batch jobs you can provide your region you can provide what kind of job do you want to run uh then the runtime version of serverless if it's a um class so you're running an uh Java or scale or then the main class name the jar URI this jar you are uh when you're running spark applications in data block which are uh using some jars or python files it's always good to have these files stored at a cloud bucket so you can give some location and then uh this is for customer image so container image but for jar then you can give foreign stored in Gs okay and then any additional arguments that you want to give all these things can be provided then the network connectivity uh you want to use networks in this project or network shared from host project [Music] and like I mentioned there's a meta store service you can also provide that right now none of it is created uh this is the one for meta store service which would basically hold all your metadata for uh say your hive um now coming to Cluster creation if I want to have a persistent cluster that I would leverage for having a history URL or having uh um the feature of interactive queries I would create a cluster so for now I would like to create it on compute engine the cluster name you have to provide the region and the zones we created Auto scaling policy right that's available here so you can choose which particular Auto scaling policy you want to use and based on the machine type you can have that um data block have certain image each image have an Associated spark version Associated Scala version Hadoop version all those things so do look out for what you are migrating from and choose the correct version you would want to enable the component Gateway which is basically access to all the web interfaces so for example spark URL your yarn URL if you want any optional components so I know I use the term interactive query a lot of time but if you do want to use it or um you want to have an interface where you can try out your queries or try out your spark commands without having to recreate the cluster multiple types you can add the components Jupiter and Anaconda and then you can create the image it is good if you understand these machine types how what memory they offer what course they offer okay so um if I talk about N2 standard four four indicates it will each machine will have four cores and N2 standard four have 16 GB of memory so similarly it means it have eight Vehicles so this is something that would help you to understand or help you uh dictate what kind of machine you want based on your application load so say uh I know my application is a compute in uh intensive application and I need at least 200 vehicles to run that application then accordingly I will select the machine time and the number of workers hence repeating again understanding your application when you're in a big environment working with a lot of applications a lot of data it is a crucial element of data engineering now um so yeah machine series and machine type is something that you can just set up if you want any additional disk size any additional local ssds you can do it and then you can finally create your cluster okay I didn't uh notice it but it's good to have this error here so whenever you're giving Auto scaling policy that we saw so if you remember the Min instances and Max instances that we had under the attribute worker config that was 10 which basically the minimum number of primary workers I can have is 10 uh just by default what we have given here um um okay um while I get to that when we talked about Auto scaling I did miss a point which is enhanced flexibility mode uh you can do auto scaling as such or there's a better version of Auto scaling which uh I personally prefer to use which is EFM enhanced flexibility mode what it does is when you have preemptive workers and they will be called off by data proc you might have some Shuffle on those workers right when those shuffles or when those machines are called of the shuffle also goes away which means I have to recompute that data and what it indicates is that that recomputation will mean more application time for me I don't want my application to suffer because of a feature that I intended to take advantage from so enhanced flexibility mode takes care of that it enables or it allows you to have all your Shuffle on the primary nodes and since primary nodes are completely owned by you they will never be called off you always know that your Shuffle uh will be there so even if preemptable nodes are called off it would not cause any Hazard to your application so we see here we have asked for two now coming back to that error on creation Min and max number of primary workers that we gave there was 10 if I just create this worker node with 10 issue 2. now if you guys remember at the start what I told about ephemeral clusters was something called quota that we can very well see here there's some quota for number of for number of workers that you can request for of course this quotas can be increased in certain cases but in general for example N2 have um I have requested 44 however available are only 22 for this quota so these are few things that you should also take care of now given the time let's go to the Q and A h okay I have a question is there any other mldl learning I'll go over Spark uh spark does not provide any um learning algorithm as such basically all the ml jobs that used to do all the for example if you want to do linear regression you can do it over spark ml so there's nothing specific that's provided by a spark as such if you read more about it probably it might be different but on General level it does not give anything as such all those things that you can run on a simple data simple cluster for machine learning you can do that over spark as well uh I have a question which is how is um how is Apache spark lake house different from data block Apache spark a lake house I'm not sure why you mentioned that but Apache spark is only a processing engine which means um it only knows how to perform some job if I say read this file and count the words that is something it can do but to do that computation you need some clusters for it to run on right so that physical clusters uh requirement that is something that cloud data provides cloud data proc is a managed Hadoop solution which means it will give you an infrastructure or an ecosystem in which you will have all the Hadoop ecosystems uh what you want to enable additional components that you want you can add those components and then you can uh fire your Apache spark job there but Apache spark on its own is only an processing engine but to process it will need some clusters that will process and that is what data proc is providing here okay so I have a question could you please elaborate more important properties for dataproc cluster while processing heavy amount of data for ML model batch processing I think we have taken care of most of this in the dataproc optimization so whenever you are having huge data you need to have your executor course executor memories tuned accordingly you have to have the right type of machine unless you are going for serverless which only offers batch jobs but other question I have for ML model batch processing so if your batch processing data for an ml model it would be huge set of data so any considerations that you would you would be doing for code level optimization cluster level optimization or even Auto scaling is something that you can use here it would help you optimize or um or more specific properties if you want to see uh like executed memory memory overhead code I mentioned then you have a hardware interval time then you have a shuffle data size partition size uh there are multiple properties which are there for spark all those properties are also applicable for uh application of spark on data block thank you I have the same question now um which adds sometimes I don't understand how to configure my cluster while provisioning for given amount of data okay uh you need to understand how much your data is so say if your data is uh see it's only 5 GB of data then your overall memory from all the Clusters should not be anything more than five you also have to consider the shuffle so say if you have five GB data and your Shuffle is only 1 GB even if you give uh anything less than 5 GB that should do but that should do the job uh in less time uh say you have 5 GB of data that you are running and your Shuffle is also 6 GB of data so in this case you might want to give a little bit of memory or overall cluster memory more than 5 GP same goes um foreign DB of data uh I don't remember for what uh exactly how many input data you should have One DB of data but it is like uh for XX GB or XXX GBS of data you should have at least one TP um how preemptive clusters are getting selected while setting up cloud data block uh this is something that is managed by data block Services itself it is not in your hand you can only mention which project you wanted for which region do you want it for and if there's any machine type that you want for that preemptive cluster this is the only properties you can configure but I think these are all you will need and based on that data proc we'll select preemptive clusters for you um so the question here what is Master node and worker node masternode is basically the main node or the application manager so if I submit a application on spark it will understand uh the application requirement it will create the requested number of worker nodes it will connect to those worker nodes this master node would have the information on metadata of all the worker nodes so if I say that I want three workers this master node will from available pool of worker nodes connect to those worker nodes now coming to the worker node these are basically some small subset of machines which can do the computation as asked to them uh it is like masternode is the brain and worker nodes are only asking what the brain is asking them to do so masternode will tell these are the computations that needs to be done and it will assign the worker nodes uh the computations worker nodes will do the computations and return it to master node the next question is do we have any restart from the stuck or uh a bent State feature of any batch processing in any case of error okay so spark is a whole offers you uh fault tolerance so that if any application is uh failed in between if there's intermediately DAC um so say um I have an application that's running but the task failed the same task can be resumed by any other task because of the dag feature of spark so to explain that a bit um you have a DAC which is direct acyclic graph which tells you what are the series of steps or series or steps that will be performed to do the job uh these would be um associate I have done transformation one and then I've done transformation two so all those series it will store so say if it failed at transformation 2 it will only resume the job from transformation to not it will not recompute transformation one but as a whole if we talk about um restarting the job from uh stuck or failed state by default any application would be tried Thrice if you want to restart the application post that it would have to be a manual trigger unless you have some automation on top of that to do that in some other way foreign yourself hello that was a great session with you thank you thanks a lot on the behalf of the analytics Vidya I would like to thank you for your time and delivering such a wonderful session I am sure of our audience founded insightful and hopeful we can conduct more succession with you in the future extensional on any deep dive into any of the topics that we saw today because it was a overview so if there's anything that people would like to have no in detail we can do that yeah sure also I hope I hope you guys have filled in the feedback poll if not I request you to please fill in the poll about the feedback as it helps us to conduct more successions if we wish to conduct a webinar or facing difficulty industry connect with us at data analytics.com the recording session of this will be available within two days on our YouTube channel also we will back with another session on 24th of January the link is in the chat session till then bye bye keep learning thank you all thank you all