Live-Implementing End To End Big Data Engineering Project With Cloud Part 2

Key Takeaways

[Music] [Music] hello everyone I hope you're able to hear me one minute hello okay uh 1 minute I think you will be hearing a hello hello hello now it should be fine I think uh can someone confirm me uh pretty sure now the E should be good like sorry e you will not be getting any Eco now yeah sorry for that uh some setting was there which was messed up but yeah we are good to begin now everyone so just before we start let us wait a couple of minutes more for everyone else to join yep uh meanwhile I will share my screen and just remove few things from my last discussion give me a second yep so few things we might have to dis meanwhile if anyone has any question please uh just ask the same in the chat we'll be happy to help yep so things are set from my end great uh part one uh shanu will be available in that let me provide you with the link just give me a second I'm also sharing my screen everyone so just let me know if you're able to see that yeah cool everyone so this is the overall uh part one video now let present my screen as well hopefully you will be able to see and hear everything fine so yeah that was the part great one and yeah cool uh that should be good I think we are good to start let me know look at your questions and answer them meanwhile just let me clear this out this is not the today's agenda yeah so let me not take up your questions everyone please keep the questions coming I hope you're able to see me today is the part two so we are going to do the project uh let me pull up the project road map sorry this full architecture cycle which we saw in the last video and yeah this was all we did great so yeah this was the project uh which we started and we did till raw data processing in ADLs Gen 2 we'll come back to this uh first 10 minutes we will just quickly have the questions I change career to do de Tas if you meant that is this course for changing career to De then yes it's the case to become a data engineer or a big data engineer right I haven't seen last live video so it's okay to continue from today uh yeah should be fine n uh we will be starting from scratch only uh not not exactly from scratch I will just give you a quick brief so you will be able to follow normally just that you should have some idea of data bricks and Spark because majorly we will be working on that today so yeah that will be helpful web development road map rine uh this is regarding Big Data engineering uh web develop it's a different kind of a road map rest and front end back end all these things I've have done that but not the right part maybe you can reach out to me separately and I can help you with that should be fine okayu pretty sure that you are able to get the link Audible and visible uh pretty sure that each and everything should be fine now uh sorry my bad now I think all of you should be able to see the uh architecture as well okay great so thanks that is good to here I'm doing a project on data does it requires to do modular coding uh get set again if you are doing it at your end no if you have to deploy it depending on what you are using it may require modular coding it's majorly on which code base you are working like I won't say that you should do a project from scratch on data science on modular coding not a right strategy okay cool uh okay so this majorly is for Big Data not really sure why you guys are asking for all the other road maps uh good thing is I am aware of all the major road maps you are asking good that someone is not asking for iOS or app web uh sorry app development road map AI road map Mora uh okay so do ml then deep learning in a lot of depth after that make sure that you understand NLP a lot then you can start with the AI things and making uh everything so yeah I'll not going to like go a lot in deep should be fine uh let us just quickly go through our course once and let me also check if the coupon is still valid so chrish chrish 10 still valid yeah that's a good thing so did expand it for other students so everyone uh meanwhile I'm also sharing so for questions which are not related to Big Data uh you can connect with me on LinkedIn and please ask them there or you can can just follow me on YouTube uh on the YouTube comment as well I reply and answer your questions so this I think is a better overall thing now let's get back to our course ones so we are taking these classes as a practice or I would say exposure so that people are able to uh answer or basically people are able to ask their doubts okay in the field of big data or data engineering if they have any so this is all about the course I have shared the link you can just go through the link via this or I think this bit. ly link is also working which is in the bottom where you are scrolling so somewhere here yeah so now let me just quickly run you through this W cabus uh meanwhile we are going to start in 2 minutes so if anyone has any doubt please ask that before I start this is what we saw in the last to last session where we started with the data source we did the data injection into ADF a z your data Factory we we created a data Lake as well and now we will be doing data transformation today okay I all good Mukesh hope you are also fine so yeah we today is Wednesday so next session we are going to have on Saturday not Friday Friday is for a little bit fun but yeah so if anyone has any doubt related to Big Data course or anything please ask that once then I will start with the overall brief about the course and then we will jump into our project great keep your doubt comings uh regarding this course regarding Big Data normal career as well just not the road map of uh web uh like web development AI or anything little out of this course but if you can reach out to me I will be more than happy to help you many students are reaching out to me for other doubts as well on LinkedIn on YouTube so yeah helping with them with that okay uh it will not be in Hindi uh okay so let me help you with that see VI so it is not going to be in Hindi but what you can do is you can just go to my channel there I will be uploading videos so I upload the video for what is Big Data it's properly in Hindi in a very good overall I would say language you can see this and I will be uploading other videos as well okay it's very high quality I make sure that I'm maintaining a great quality in my YouTube videos okay so yeah that is the thing but the V the course it's going to be in English Okay but the English will be fine like the way I'm talking everyone should be able to get it if you have any other problem being Hindi my other language which I can speak I will shortly be see my aim will be to make sure that I'm able to explain you okay now with the help of chat GPD even if it takes some other language I will say to it in English it will translate into that language so so I don't think you will have any problem with respect to the language you can also follow my channel there I will be putting content in Hindi sporadically not like all together but yes this course will be in English Okay and many students are asking for udmi course as well so udmi we are recording and it will take time so if all of you have seen till now this is the cabus right it's a lot huge cabus and we will have multiple courses there Chris sir has already clarified the same thing as well so it's going to take time the videos are not going to be the same which we are going to take in this session no we we are re-recording and making sure that each and everything is up to the mark to help even a fresher start into the Big Data journey and yeah master all the clouds as well so AWS Azor and GCB we are targeting all the three clouds okay uh baljinder if you will go on Krish Nayak Channel this channel which you are looking this video on you can go to the live the last three four session live sessions are the one which we are discussing till now okay uh Cristiano Ronaldo 7 uh easily 7 to eight months minimum if you do things in depth and you understand and you make projects which are basically having many components just like this one which I have in front of you so yeah that is going to be good Should I stick to ml or should I dive into gen first do ml then only you can just dive into gen okay before that anyway it's not possible okay M so I hope that is helpful uh yeah I think most of the questions are answered if I miss this session will the same thing be covered in the live by starting from 21st uh I of in the live class I will be explaining things in a lot more depth and the project part will come afterwards so see right now I'm making sure that I teach you project but still students will not be able to get a lot of things so just to give you an example uh just give me a minute I will be teaching things in a lot more depth in the course as well so you will not be missing out on anything as such just that the project part will come a little late in the live class like we will make sure that we understand Azure first properly right so this is I am on your Azure everyone pretty sure that all of you have seen uh spending some credits for you all and have deleted all the resources okay all the resources are deleted because I don't want to incur cost so yeah I think we can start with this just a second meanwhile just let me quickly brief all of you about the course so this is the course which we are starting everyone Big Data with Azor and AWS Cloud Mastery I'm sharing the link again so the course is going to start from 21st December okay so 21st December this one minute let me just okay yeah not able to write with the B just a second everyone yeah it's fine great so yeah let me just quickly help you all with all the details so we are starting from 21st December that is I think now next Saturday timing will be 8 a.m. to noon right so 3 hours I will be teaching 1 hours dedicated I have kept for doubts if the doubts are still there I will stretch it maybe but I will make sure that each and every doubt is clear so you will not have to worry on that I always am very receptive to doubts and it will be in a one-on-one fashion then proficiency we are saying professionals uh cleared it many times if you are someone who is in college who has never done coding please don't join this course uh first get your hands on coding you can follow my channel you can follow Chris sir Channel you can follow anything but first get a taste of it field what exactly uh your technology and Tech is that thing is very important if you are someone who is let's say half one one and a half years into the job has a little bit of idea uh feel that his career is going to a data engineering side or would like to get that first reach out to our counseling team and then join the course if you want if you are someone who is 3 four years who is doing data analyst job who feels that their career is stagnant they want to enhance or basically accelerate their career you have the idea about the tech two years or plus I think this course will be going to be perfect for you so again fresher fresher who are basically Al in college please don't join uh data engineering is something which is a advanced level skill it's not a not that it won't be helpful like honestly I wrote and learned few things in my college only as part of some research work which I was doing but data engineering is not a beginner level course as such it's a professional level course only okay knowledge helps of course every knowledge is going to help but first learn about Python and other things that is going to be helpful okay so that is just to clear freshers in job still uh it's very right for you just make sure that you make up your mind because you will have to spend some extra uh hour or extra time at your end okay we are making it a lot clear I personally will be teaching each and everything from depth but few things I will be making sure that you have an idea about for example python we will be giving you with the recordings not teaching that in class databases and SQL as well right few things like networking I will assume that you have a basic idea about it not a deal breaker but yes if you are someone who doesn't have any idea about anything any coding no coding experience uh please first get that and we I will be more than happy to help you even after the course is finish or even if we are in Midway of the course okay so yeah that is all about the timings and everything now about the features everyone so dashboard access with one and a half years then Community chat forum for discussions uh that has already started I am answering are lots of Doubt there students are coming up with their personal query and request and everything so that's a very good thing because there we will be also sharing if we get any job openings or normally you will be able to uh normally interact with your peers as well so that is very important honestly and we will have hackathons as well so I will make sure that we have some projects and stuff that is normally helpful to uh just start or basically in your career in your data engineering Journey if you have someone or if you have team with which you have done the projects that is majorly a lot helpful Live diet clearing after session so doubts shortly will be cleared that is not a worry hackathons with reward and job referals if you have any and resume discussion and mock interviews so I have actually taken lots of interviews I think easily 300 plus or more honestly uh with scaler coding ninjas in my all the jobs I like the interview process a lot like taking interviews giving interviews so we'll be sharing all my experiences and yeah that is all okay you can reach out to this number if you have any queries to enroll you can just go to this link as of now uh you can apply chrisan it is still valid and the course we are taking minimum like some fixed amount of students not like it will be a 2,000 3,000 batch no uh we want to make sure that people are joining and they actually transform their career that is the major aim honestly that is the vision with which me and Chris are making this course and we have designed the curriculum and yeah this is the cabus uh one second yeah so this is the stabus everyone uh the whole walk through again not going through it uh it will it is there I have done that in the last four classes so you can just see those classes once to get an idea that what all we will be covering Okay cool so yeah let me just quickly have your doubts and then we are going to start with part two of our project today uh CR7 Christiano Ronaldo 7 if you have a knowledge of these things it is going to help in your career let's say your data science ml career normally you will be majorly working as a software engineer and then you start with these things if I be really honest so data engineer is a subpart of software development on engineer that is how I got my exposure to data engineering if you have that knowledge and if you basically share that right with your team it's going to be helpful there is no such thing in a way as fresher data engineer job as such it's just that they make uh let's say they will hire you and if you are let's say in the analytics or database team uh in your overall organization and companies have them a lot like there are many teams which are dealing with that in every company okay so it's where your data engineering Journey will start because it is something which you learn and understand so as a software engineer you will be starting your journey in a data engineering field that is how it happens in a real sense okay great currently working as a mean stack developer I pretty sure gopal it's M Stack not mean stack okay should I jumped into big data or data science or it's fine to uh gopala that's going to be your call if you love uh making apps if you are absolutely loving your work data science or data engineering might it might give you that not might not but that call is for you uh if I talk about in terms of the opportunities then yes uh the opportunities in the field of B basically this big data engineering is uh I would say increasing day by day because of data getting generated so from that point of view it's a good overall investment and feied but yes if you have to do that or not that will depend a little bit on your personal choice as well okay uh Prine when you enroll we will be uploading the SQL video next week this week we will be uploading the python videos okay uh course completion certificate yes certificate will be there can I list skills I'm learning on my resume for sure uh you will be able to list Azure AWS all the technology if you will go through the course structure you will see how many things are there so you will be actually having a lots of experience in many Technologies and I can assure you this that your CV will surely be a different allog together once you covered this course and how can I say that because again I know what I'm teaching I have worked in these things for years so yeah your CV will surely totally transform and everything you will be able to enlist on your CV along with the knowledge see my aim will be that all of you also have the knowledge and the confidence that is the major thing I will aim right CV to you can even just see that give it to chat GPT and it can give you with the CV but knowledge will also be there along with that you will have some great awesome projects we will do projects which are 4 four five five hours in length right very near to what you do in your job so you will get that exposure as well and in different clouds so you can think that how good it's going to be okay yeah so that was Raven question will you be able to Big Data related certification uh AB again you will be getting lots of that ex knowledge and I'm pretty sure that with that you should be good to have uh any certification just that what might be required is if it is a very specific kind of a certification you might have to work a little bit on top of what we all discuss but again uh my like this course we have made up with a mind of something which will be useful in the job in cracking the job and everything so not with respect to any particular certification or such but yes knowledge will be there after that you are good to and for sure that will be useful as well so that's the real honest opinion uh for the certifications honestly I normally don't run a lot behind certification so yeah that should be good okay mean stack sorry my bad my bad oh sorry my bad my bad I I forgot I thought that it's a type of M and uh like reacted that sorry my bad yeah only Messi is left now into data jobs yes exactly e e uh hello everyone extremely sorry uh face some issue just give me a minute [Music] uh hello to hear me still extremely got some issue with the connection let me share my so yeah just saying that you can follow the series as well for python and yeah now let's I think it's good to start with this particular thing now we have enough queries and thing so that's normally what first 15 to 20 minutes are for but [Music] yeah cool everyone I hope everything is fine just to have my screen as well yeah so this was the major architect diagram everyone which we did in the last first part of our project I hope each and every one of you remember this yes or no I hope that is clear everyone so tun I was answering your question got some issue uh this is the prequisite we we have a module for this right uh you can go through this we will be providing you with the recordings everything is mentioned in the course right you hear EOS 1 minute uh sorry now I hope it is fine uh very bad uh from bad from my end but now I hope it's fine uh just add some issue now everything should be fine uh not sure why I get the internet issue at this end from AEL but yeah cool so great uh now yeah everything I have covered with that uh now let us jump back to our project so we have done the project the first part so people if you remember we started by getting our data via a your data Factory into our data source which was ADLs gen2 which is data Lake right now what we have to do next is we have to work on our data now comes the RO role of data transformation okay so it happens that as of now I have just got raw data into my system right everyone so if you attended the last class I'm going to give you a quick brief as well we had the e-commerce data okay so let me show you the data as well just give me a minute hub yeah so let me just go to the data everyone this is my GitHub repo Big Data projects eCommerce and yeah so this was the data customers order details orders and products right everyone so this is what we saw in the last class that how going to Raw going to Raw we were able to get this data across into our aor data Lake right so we created this ADLs Gen 2 uh Ahmed krisher right now is on his way back to India he went to us for the AWS invent event so I think he's still on flight or might have just landed today maybe some hours back okay I hope that helps great so let us go back to this everyone uh I'm going to create a new Resource Group again uh I delet everything and that's a good practice overall if you are if you don't want to incur cost okay normally I made sure that because we will be having some time uh everyone I hope I audible can anyone confirm me once if I'm audible or not hello everyone uh ran uh request you to check once please okay please uh like these things these technical issues okay everyone just a last check uh please help me to confirm three things my audio video and the screen you are seeing uh it should be 1080p quality so we have purchased a good software to make sure that you see in this can anyone just quickly confirm yes 3s or something so that I'm sure and we are good to continue don't want technical issues to have any problem it should be 1080p you should be able to see each and everything properly I will zoom in as well but yes majorly you should be able to see yourself only because we have 1080p but yes it's 120% now Okay cool so thanks a lot uh Raven uh please make sure that who's the person yes Ren and everyone else please make sure that from the technical side you just first check your end and then let me know because for that I have to stop the flow Okay cool so we are making free trial we are creating the First Resource Group we are making it in central India so Microsoft Azor has three data centers in India one is in Pune which is the central India one I think is in Chennai and there is another one which I forget where it is okay so let us make e-commerce live part two okay or let us try just e-commerce live if it is available we can just go to next uh name and value we don't have to give the tags are just for the way you tag thing right in your notes app or something to have some distinction or basically get them quickly don't want that just create this okay so we are creating a resource Group I explained that in the Ed I will discuss after for data science afterwards uh we let's we keep that towards the end okay so I've have created a resource Group everyone in this now I will be making all these resources so azard data Factory ADLs Gen 2 azard Data bricks synapse all these things will be done in the last class I made sure that using aor data Factory we were able to get the data into our data Lake okay so this is something which we made short of now let me do one thing let us create storage account okay create storage account so here I am creating this storage account again I am not creating a your data Factory again reason I deleted if I would not have it would have just in basically had some cost which I might have to pay and it was I think what six days back so yeah don't want it for that cost okay Ecom data life yeah the okay it must contain just lower characters it should be uh overall unique I think ex uh yeah so it should be unique the name must be unique across all existing storage accounts in Azure okay so this is your proper data Lake in aure where you can store your data it is stored in a blob format BL O Okay so it is stored in a blob format now if I just scroll down Central India is the region okay primary service we can just uh keep it empty we are going to start ADLs Gen 2 ADLs Gen 2 as your data Lake storage performance standard uh redundancy we want local redundancy so there are multiple choices if you will see here okay local rency is the lowest cost it will just have my data across servers in the same location okay okay so that is why it is local goo is provide you extra benefit the way you can think of this is that uh you will have let's say in India one copy will be kept in Pune and another will be kept in Chennai if tomorrow there is some problem in Pune location some natural Calamity some natural disaster your data will still be safe okay Zone again we have different zones which are created okay we have different different zones which are created it will have those within zones right so if I remember right uh for example I think for New York it's I think in AWS the zone is San Francisco so they are parall like opposite so majorly it gives you a more better data security basically you will have a good redundancy so we are selecting lrs more you will select more the cost will be okay so just to explain you in a very easy terms let's say you have a very good uh let's say this is your one minute yeah so let's say this is your house you have a very good say copy of something or you have an iPhone you have two iPhones first is you are keeping both in your house so that is local dependency okay if a thief breaks in or if there is some Calamity both of them will get destroyed okay so that's the first one second is you are giving it to one of your friend who lives in India somewhere okay so who lives in India somewhere so if tomorrow you have two iPhones you gave it to him if tomorrow there is a problem there is something some natural Calamity a very big wave okay yours will get destroyed but you will be able to use this okay extending on this knowledge let's say you have a friend abroad so if tomorrow there is a overall some failure in the Indian data centers maybe due to anything still your data will be safe so this is the easiest explanation which I can give to explain you about r okay great so let us move again forward so this thing is very important everyone enable hierarchical name space this we can ignore on for the timing this is what gives your data Lake have folders okay so just to explain you again what happened is that let's say this is your data Lake overall it's a store for storing all your files and everything right in this what happens is that if we make it just so we have two options it can either be blob that is the first option or it can be ad LS Gen 2 this overall choice is very important this tick is actually the difference okay between making it ADLs gen2 and a blob and what's the different exactly let me tell you let's say you have a laptop you have a desktop in your laptop right so for example just like I have uh this desktop okay I have this desktop I can create folders here okay so I can save my files by creating folders this is the option which helps you in creating folders and that is why it's ADLs Gen 2 in this you will be having a data Lake where you can create folders this is actually the option which is very important to tick if you want to make a data Lake and not just a simp simple blob storage I hope that is clear anyone has any doubt in law what all I have explained so I have explained you two very important things in your storage account so we are I will just go back to the diagram once again this is the diagram we are creating storage account again okay not creating uh this as of now again as your data Factory we are not creating I'm creating storage account again and then we are going to go to data bricks to work on that data okay so let me now move forward other things we can just keep as it is network access it is public again it will take a lot of time for me to explain each and everything right now something which we will be keeping for class most tech most I would say things which you have to understand I will be explaining but right now let us go forward we don't have to change that should be fine okay encryption tags and we can just review and create so it's creating now how does Microsoft a your differ from AWS services in terms of cost V you will have to go to AWS once uh it also gives you the free credits uh you will have to see that cost okay each and every platform has its own cost we will be trying that we use the uh available credits only right that is what going to be our Aim so you should just have two I think two or three credit cards I don't think it's a problem like you can ask your friend or something it is not going to charge you it is never going to charge you unless you give it the permission and it is not that easy to give as well okay so for that you just have to follow the instructions and you will not be charged so that thing is for sure okay great so see everyone we have the resource with us we have created a resource in this resource now I can go to container so see here also we have four options if you will see here this is the major thing which we are concerned with data storage so we have created something to store store our data right so what are we concerned with data storage we can create containers we can create containers that is the basic and the most major use for which ADLs is used file share is something for sharing the files cues is something for streaming data many of you might be aware about Kafka a very common technology for streaming service streaming data is just like realtime data okay and we can create in the plain old table format as well we majorly use containers it's a blob storage which is AWS own and we will be storing our data in that way I hope that is clear everyone so I'm going to create a container what's going to be my container name Ecom data life so I'm trying to keep it same only let's see if it takes it okay now let's do create and it's cre creating it okay uh LinkedIn user for doubt session I will be sitting and answering your question one by one uh again I am very much serious about each and every doubt which students have so if there are a lot more than we can have let's say every two three weeks we can have extra session where just doubts will be solved but majorly I prefer that once you are understanding something then and there only after that class you should have your doubts clear so that is my major concern so for that I will be sitting unmuting you one by one and then answering and clearing all your doubts apart from that uh I will also be clearing the doubts on the fly on the community and other people can also do the same thing but yes that is going to be the major thing but yeah if you are serious about the course then I will make sure that each and every of your doubt is cleared okay so after the class itself so because that I believe is the best time to clear any doubts great let us go back one minute so Ecom data life now I am inside the container you see this add directory everyone do you see this add directory button add directory you remember I clicked that hierarchical data hierarchical structure hierarchical structure when I was seeing that this is a very important thing when I take that you won't be seeing this if we would not have done uh tick on that you will not be able to see this you cannot make directories then okay uh wish love brocher you can download from from here this is the link so vishav uh you have to actually go through so you see this vishav the one this thing this link overall just go to here if I reply on this via uh streamyard from which we are streaming this you will not be able to get the reply on Linkin I see that you have asked the question bya LinkedIn so on this post from which you are seeing this live you should see this browser and everything cool so yeah everyone so this is the major thing input save I'm creating a a new directory output I'm creating a new directory in the input what I will do so just give me a minute I will just add all the data this time directly because I am not creating ADF again for that for creating and seeing the a your data Factory you can just okay my bad not like this upload and now I can dag and drop again my bad no worries so downloads and we have what do we have one minute everyone just a minute everyone yeah so rower files pictures and this is the data which we last time we uploaded this data via overall so you will see blob type block blob block size 4 MB access tier hot so I will tell you what this access tier hot means uh this is for the data which you will be accessing a lot right for example uh let's say this is maybe what you can think of is this is your phone storage where you have the decent photos your school life photos or your earlier childhood photos might be somewhere in some album or maybe in your home PC or somewhere okay so we have three access tier majorly hot cold and archive uh we will be using hot majorly for the data which is getting used a lot okay blob yes uh V exactly so it is the AWS like hdfs we have this and we can just upload again uh see everyone many people might not be aware what hdfs and everything is these are all the theory part which we will be covering in course let me know if I can take a session for that as well but yes for the timing you can just think and keep understand them from the easy example which I'm explaining okay so now what I have done I have done till this part again so this was just for a quick brief of because many students were there who were not uh who have not attended the last class and now you must have seen that we have gotten our data earlier we got this data via HTTP request HTTP via GitHub and I showed you that you can use a lots of things Amazon S3 Amazon all the Amazon service Google service SQL DB no SQL DB each and everything is available to you okay great let us now move forward so we have this data here everyone now what we are going to do let us now go to Azure data braks okay here again I will have to create another Azure datab break service so let us do that Azure datab break service Resource Group we created one e-commerce life workspace name e-commerce live only I'm keeping uh Central tier it can be premium should not be a issue and let us do manage Resource Group name also e-commerce live so I'm just trying to keep it same only you will see the reason afterwards but yes okay everyone so I in the last class in the last session okay we discussed great length in data breaks and Spark data breaks I showed the Community Edition and I showed that how it is very straightforward and equivalent in Azure as well so let me just move forward no things you have to change here no problem no tags review and create and it's going to create this now okay and we will see that it will take a little bit time and it will create the service so let me now explain you a little bit on this what happened uh there might be at least one resour deployment operation fail please list deployment operations for details okay some issue happened let us see what happened exactly um okay bad request why is that okay so we have given the name which is same everyone and that's the problem so you can see the manage Resource Group cannot be same as application resource Group which makes sense so let us do one thing let us delete this let us delete this deployment all together let us go to data bricks again so it's going to get deleted in the back end let us create another service so it's saying that okay you are fine you can select this Resource Group that is all good okay here you cannot have this so what you going to do data bricks okay e-commerce like and here we can have workspace as uh Ecom life okay so workspace name I'm giving Ecom Life Resource Group we have already come uh created in the starting and manage Resource Group something which spark or database is going to use we are keeping it on this name I can just directly do review and create I don't have to do anything here next next no need for that I can just directly I'm not sure why it doesn't give an error when it does the validation here should have given it here only but yes so that's kind of a bug in aor I think right now let it create cool uh now let me explain you a few things which is going to be very helpful everyone so what happens is that we have data bricks which I discussed in the last class that how it is created by creators of spark how many of you remember this thing data braks it is created by creators of spark right aor says that hey I will be giving you the hardware so I will give you all the hardware whatever you need and we will see that going forward so this is provided by aor the easiest way to understand this just like you have a laptop you can take a laptop from Lenovo HP Dell okay that is the hardware so you pay those guys for your Hardware data breaks provide you with the software so just like Windows right just like Windows okay is paid and yes Windows is paid it is not free you might be using a crack version or it must be already installed we have to pay aor for the hardware and databas for the software and why are we doing it because datab brakes as a software is a very good technology it has been created by creators of spark and it's a lot optimized in many things okay so now what we have is if I just explain you guys with one more thing let's go to sticker so now we have a overall Space 1 minute let me get that diagram once again so pay attention everyone now what we have we have this overall device with us okay we have this thing here we have data brakes on top of it which is an external software which is not something which is done by Azor and one more thing to understand here we have this a ADLs as well we have this ADLs as well now what happened is that and this is very important to understand everyone this is on Azure and this is on datab brakes okay this is on aor and this is on data brakes okay if I need to access data if I need to access data I cannot do it directly I will be needing I will be needing some key or some permission so that I can access the data how many of you are able to understand this thing this is from datab bricks this is from Azure to read the data from this Lake from this Lake which we have we need some extra permissions some extra security is mentioned here for that we need to have some overall permissions which we will be doing okay so that I can get the data from my data link anyone has any doubt in this thing let me know okay so deployment is still in progress it takes a little time meanwhile let me know if that is clear everyone is it clear everyone anyone has any doubt in this let me know in the chat just to be on edge didn't get exactly uh everyone so I will just uh so deployment is complete let us go to Resource okay so see let us launch this workspace just like we were launching workspace till now let me launch this again and now let us go to data bricks that's is your data braks see it is doing this thing starting your server let it go forward great so a lot similar to what we saw in the last overall I would say database class I showed you that time as well this particular thing you will see that we can and SQL here machine learning is also supported data engineering we have inje Delta life tables everything but our aim is to create a compute first that is to get the machine okay uh Aditya I will be covering each and every of these things in the course uh so you don't you will not be requiring uh I don't think it will be required if you have time you can do that but a your and all that Basics I will be covering properly okay as of now uh we are just is doing a project so as you can see right it's in one and a half hours of course no one can teach you all the basics of aure all the projects so right now that's why I'm explaining you things as they are coming in the course we will first understand thing and then basically Deep dive into each and every of these Concepts okay great so let us create a compute everyone now in on Azure data bricks okay so it's unrestricted you can see this we didn't saw this mult node cluster option right we just saw a single node cluster because it was datab breakes Community Edition how many of you remember that thing uh LinkedIn user I will be coding in pisar just as soon as I complete this thing I will be coding in ppar only after that okay so just we let us quickly do the compute part and then we will be coding in ppar so understand everyone we are we have this way through which we can have the worker and the driver we can create a single node machine as well we can create a multi- node cluster as well you will see that here you have machines again which you can select multiple machines are there as of now I'm going through single node only not a problem I will terminate it in 10 minutes of inactivity so they charges you based on how much you're using the machine okay so that's why I am just making sure that if after 10 minutes it switches off itself okay okay so this is the all thing everyone which you can see in front of you I am creating a single node cluster okay let me just do it change the name single node cluster 10 minutes so this 10 minutes I normally right to tell me that okay how exactly when it will turn off runtime is basically this I think is Ubuntu only so Ubuntu version it will install Linux on those servers it will physically go and get me the machine this is the Scala and Spark version then this is the note type uh we can just have a basic one only okay let me just do this we can actually have this basic one only I hope it is present in this Central India location and I can just go and do create compute and now it's going to create the compute for me what exactly will it be doing here it will physically go it will say that hey someone has requested a machine with this particular thing hey someone wants that this is the ubu version and these are the spark and Scala version which should be installed please get me that thing meanwhile Azor is like hey mayang just wait for maybe 5 seven minutes what we are going to do is we are going to get you the resource we are going to install Ubuntu spark everything there but you will have to give us time okay meanwhile it takes some time I was explaining you in the diagram that see this is provided by data bricks this is Pro provided by data bricks this is provided by Azor for Azor data bricks we are just using Hardware by aure right so for this particular thing what we need we will be requiring a password or authentication okay everyone so we'll show you the process don't worry but I'm just tell you exactly what is going to happen so we'll share that so yeah that will be good uh kichu I think you for you didn't attended the last class so the difference between a multi node and a single cluster is something like this see this is your single note cluster okay this is your just single note cluster what happens in your multi cluster is something like this we have this particular machine which let's say it will be the master or it will be The Driver node and along with that we will have multiple of these workers who are going to do the work so this is the very Basics about distributed computing in general explained that in the previous session here we are saying that hey I have just one machine uh this is what I'm going to use for the computation and everything in your multinode this is how it happens so these are your workers and this is your driver or Master as the name suggest driver in this particular architecture right it's a distributed architecture for heavy workloads majorly we use the worker machines this driver worker architecture again if I choose that the cost is going to be high and if my data is something like this plus one more thing to understand while coding there will not be much difference okay if you're coding it there will not be much difference so when you choose this kind of an architecture your aim of driver is to just tell your workers what it has to do now datab break spark everything is optimized so that you will have to write your code as it is you don't have to worry on anything but internally it's going to work on different different machine okay so let's say if you have to uh maybe let's say you have to do some joins right it will it is going to divide that thing it is going to spread that across different machines if you have to count the number of words let's say in a very big file it will divide that file and send it across these machines so that it can be done pretty quickly easiest example to understand let's say I give you a book okay or let's say I give you some pages and hey count me this number of pages you call your three friends and you give them each some some part of your pages okay so I give you a very very very big book it has multiple of these Pages what you do is I ask you that hey can you count this you being a smartman what you do is you call three of your friends you give each and every one of them some part of this book without counting and you take the last one in this overall driver normally doesn't do any work it just guess like tell the workers what to do that is what multinode cluster is but let's say your friends are going to beat you if you just give them the work so you are also taking some part on yourself now if you will count these pages right it's going to be a lot quicker I hope you understand why because earlier you were the only one working on them now you have distributed Computing but one more thing to understand one more thing to understand if the work is very less let's say if you have just five pages now it is not that optimized always because just counting the five pages maybe you can do it yourself why will you go and talk like first you have to call your friends then you have to explain them what you have to do right so that's a tradeoff always that it is not that always for small task also spark is going to to be faster okay so I hope now it is a lot clear in the easiest language possible so kichu what was the name yeah kichu I hope that is clear okay so are we Distributing our task yes for instance a day frame can be distributed into different machines yes kichu that is exactly the whole idea behind distributed computing and Big Data as well when working in an industry project can someone access our cluster how is cluster sharing managed uh Vishal no one can access your cluster provided you have proper security there for example you cannot even uh access this cluster as well which I'm giving okay so see computer is running everyone what we can do we can go to our workplace create a new notebook so notebook is getting created let us connect to our cluster single Lo cluster and yeah we have this okay how does the distribution takes what is the criteria so LinkedIn user for that it's actually the very basics of Hadoop and map reduce architecture how that happens I will not be explaining it uh right now the reason is that I would like to make sure that it is properly explained so the criteria is majorly the same uh just to give you a basic idea okay I'm giving you the basic idea not any architecture diagram or something your driver note here it has the whole idea that what exactly is happening it knows that okay which data is kept where okay where the copies are also created so replication is also a very good factor in your overall work of your distribute this architecture it basically says that okay let's say you have to count Pages count some things and just the data is kept here it is just going to ask these two workers that hey you have this data I have a request to work on this data please do this work and get me the results once this happens so the doing this work is map operation okay in a way it is map operation once your results are there then we reduce those results into a single results so you will have two results let's say he says that hey I have 50 pages he says I have 40 pages we will reduce this to a single answer which is 90 this answer will go to your master driver and Via that to the client so some client will be sending out the request right in a very easy way that is the way it happens now there are again many things which comes into the picture uh the block size okay uh overall resources and all these things how they are shared kept how many tasks are created all these things are there which is something which we will learn when we Deep dive into spark spark and map reduce majorly okay so pretty sure uh it would have been normally easy and easy to understand so we can now see just would like to show you notification no I don't want to see any notification and we have spark here as well everyone I hope all of you are able to see this yes or no can you just tell me now till now anyone has any doubt anything which is not clear and yeah I have been speaking since a long time so let me just quickly have water and if anyone has any doubt as of now please ask that in the chat I will be more than happy to clear them so next part what I was saying is aure aure data bricks to ADLs Gen 2 see uh there's a proper tutorial which is given by Microsoft itself okay I can just tell you the basic idea behind this we have to create a client secret okay for your service principle we have to Grant the service principal exess okay we have to add the client Secret and then we have to configure these properties so we will see many properties yeah these are all the properties which we have to configure okay uh yes kichu so yeah that is basically big data and distribution 101 for you what we just discussed so what all you have understood that is the major idea the driver manages all these things driver maintains the metadata okay so driver maintains the metadata which in which it knows that okay what exactly is happening I hope that is clear again anyone any doubt anything not clear pretty sure that uh the way I've explained now it should be lot clear any doubt before I move forward anyone will be good if you clear it night now why I'm doing what I'm doing again these are not very easy things to do honestly these are little uh complex so if you have any doubt please make sure that you ask that good to move forward can I get a yes no in the chat something uh ajg what is data breaks for that I will suggest that you see the last video I explained in that properly uh what exactly is data breaks so what we can do we can go to YouTube okay minute yeah so we can go to YouTube we can go to Chris Channel Krish naak we can go to SAR Channel we can go to live and see introduction to data brick and Spark uh around one and a half hour class in around 45 minutes I must have explained this what exactly is spark what is data brakes that is going to be helpful okay so yeah now I think this is clear to each and every one of you that what exactly is happening we have this ads Gen 2 we got the data here we created our datab BAS cluster on a your and now we have to work on our data okay clear everyone so I'm moving forward if anyone has any doubt just ask that and I will do that once we take the next break right so for the timing just let us copy this code given by this only so we are pasting it now you will see few things you will see few things here okay and I could have done one more thing just to show you how awesome is it see uh give me the code to connect to my ADLs Gen 2 folder to get the data let's see what code it will give it's going to be straightforward easy how good AI is let us see so it is saying that ADLs path spark. read format uh okay so one second let us just change it a little bit give me code to have permission and conf set to connect to ADLs Gen 2 okay now let us see this so yeah see now if you will see everyone this is I think it's the same code right how many of you can see it's the same code which AI is giving uh vikrant uh data Brak is also there in every of these clouds so be it your AWS gcp or your overall I would say aure the hardware is given by these services and it's the same okay so you will see here everyone this is a generic code right this is the normal code which is given on this website you will see many fillers here see storage account okay see how many places storage account is there uh application ID then directory ID it is asking you for lots of things right why is it asking you this I explained you again a your datab breas is a service given by this data brakes is a software of data brakes uh it is not directly in terms of Azor whereas Azor ADLs Gen 2 this data lake is within Azor so we need to have this permission we need to have this authorization okay so let us move forward then let us do go to our storage account let us go back towards your minute uh kuu that is using gen model for giving you the code nothing else so that is just like any other gen uh where is a storage account so Ecom data live is a storage account everyone as you can see right okay let it load let me just check yeah so we have the dates okay so one minute let me refresh it let me just go back so storage accounts uh this is the storage account where we want to have this we can go on IM Access Control add R assignment okay so before doing this actually we will have two 1 minute I'm not sure why it is working very slow minute let us go back so we will have to get all these things everyone so I'm just trying to show you that how we can get these things okay just a second uh what we do social yeah accs key is my bad so it is just hanging a little I think you can also see that it is not uh working very quickly right just a second this is just giving us some things mean ask your doubts everyone not sure why it is taking us some time let me just get this storage account name is something this so let us go ahead storage account see everyone we can just change it everywhere okay so I'm changing my storage account here everyone see okay so pasted it everywhere where else it is needed I think yeah storage account everything is done store storage yeah so it is done everywhere I've changed my storage account now let just see storage account application ID okay so I just want to show you that how exactly we will be getting all these things okay we will be seeing that how exactly are we going to get these things just give me a second see is access key yes in a way key yes so you can think that they are similar only okay so uh yeah I think they were let us just go back let me close this if I if we go on our storage account Ecom data live so just that as Y is working very slow not really sure what happened here but yeah internet seems to be fine uh not an issue with that but yeah uh okay so why is it not working 1 minute everyone so it is just taking a little bit of time cool so let me do one thing actually meanwhile what we can do is we have this workspace okay so let it uh run meanwhile everyone what I'm doing pay attention so notebook query more if I go get folder yeah so add or uplo data I'm meanwhile using the local one uh zor is like normally it happens in free account sometimes okay so what we can do is create a table from fil in ads using not this we can just create we can update the data here let me just update the data here only okay so what I'm doing everyone just to make sure we are just uploading the data for the timing okay I will just check that as your is just taking a little bit time and it happens in the free account sometime don't worry so we are uploading our data here okay so just that we are using this locally okay just local as your data I explained you that exactly we will be needing to have uh this thing so let us just upload the data majorly we we connect to this using the code which I showed you okay so let is just create the table okay so I think it has just did something wrong let me just upload them one by one not sure why it is working very slow just give me a minute let me quickly check meanwhile if anyone has any doubt please ask the same it'll be helpful to clear that okay why are you so slow yes I want to leave let us up it one by one okay so I'm creating the table so in the data engineering we just going to data injection and I'm uploading them one by one uh not really sure it is not working very Nic see if I go on e-commerce data overview okay let it load subscription ID and everything so we needed the application ID and all these things right so let me go to security networking access keys yes I think something has changed here in this wait second shared exess signature so one minute everyone we will do that meanwhile let me just upload the data here we have customers now let me go to data inje okay we can upload to dbfs as well so one minute yeah cool so we are all this data everyone just give me a minute yeah so all of this is uploaded now what we can do is we can go to our workspace uh actually we were working on our notebook right so this is The Notebook we were working on and we have this eom workbook okay so just let us quickly check if we have the data or not with us start attach and then okay so I think it has been 10 minutes so the cluster got terminated let it just start again okay so the data bricks can do the V process of ETL pipeline uh no kichu not really we cannot get the data from different things it supports a lot of things here lots of uh inputs but not the whole pipeline cannot be created and if you just pay attention many things are there right which we have to connect with them then uh many connectors are there in the ADF right which is of course not supported so this I'm getting the data because Y is just working a little bit slow but yes majorly you cannot do all these things that you can just get the data and everything from this okay then it attach to the cluster everyone if anyone has any doubt meanwhile just quickly ask and we'll be happy to answer that great so let me just try to load this data everyone the one which we have uploaded so what where it was was exactly let's say if we have the orders orders data right so we have spark session one second we will have to get this so from P spark. SQL import spark session okay so to get the data which is which we have just uploaded right spark session dot then we will have to do after that builder. app name and let's say I'm just giving an app name Ecom right so I'm just giving this as an app name data break I think yeah all the doubts are answered so and get or create so this is just to create a new overall spark session everyone right this will create a spark session and I will have spark here something which I'm going to use for this let me just go back and Spark okay now I have to read the data right so orders data okay is equals to spark C it is giving you this V code right spark. read CSV file store tables order right and we can do this let us try to read this if I remember okay we will have the data in not in tables but in that so let us just quickly see a spark job is starting and it has read the data let us just do one thing now orders do show and we have the table with us order is not defined my bad orders data orders data and you will be able to if I run this cell you will be able to see this particular thing right so we are able to see the data everyone right in a similar way we have four tables so customers data equals spark do read do CSV customers okay next was we have this customers data then we have the products data it is just helping you to fill in the code thanks to Ai and orders detail data right spark dot so I'm just reading the CSP I'm just reading the CSV and once I run it everything should run and we should have all our data this is what our aim is so we are able to now get our data okay everyone now just to show you what exactly is happening behind the scenes so we have this data with us now we can just see it here now pay attention this is the schema right so the very first thing which I'm going to do everyone the very first thing is which I'm going to do orders and customer right DF equals so pay attention orders and customers we have customer ID which is the common thing here right everyone I hope everyone is able to see this we have customer ID which is common between them I have the orders and the consumers so a customer must have made an order right so what I can do orders uncore data do join C do join customer data orders Data customer ID equals customer data. customer ID so all I'm doing is and I think I have forgotten this T it again customers customers okay so pay attention everyone and now I will run this oh my bad which is it customer underscore ID so AI is failing here and here as well we have customer _ ID let this okay cool uh one thing more to notice here everyone if we if I have done this particular operation do you think we have the results as of now how many of you have heard about spark lazy evaluation lazy evaluation of spark how many of you have heard of it lazy evaluation Krishna I didn't has customer not K it was C Omar so I forgot the T there right but no worries we have the data with us now so how many of you have heard about about the lazy evaluation of spark everyone what happen is that when I'm doing this right when I'm doing this particular code okay what exactly is happening is spark as of now will not be calculating or doing this thing it will not be doing any particular thing here it will wait so it's made a d a so I will just explain you just let me show you that uh spark Spark transformation and actions okay again these are all very uh I would say technical things which again we are going to discuss and we have more than like two three weeks of classes for this particular thing what happens is that there are some operations and Spark which are transformation some are there which are actions okay Sparks create a dag direct ayic graph so let me explain you yeah see pay attention so see you will see that these all okay these all are the transformation these all are the transformation as you can see in these operations spark doesn't do anything it just say that hey I know that I have to do this thing but I will do it when action is required okay so it's the same as you know that your exams are coming you have that knowledge but you will start studying once it is nearer right everyone so we don't start studying like for example I know that okay I have an exam in November maybe or let's say next April I will just start my action once it is asked for me to do right in a very similar way we have Transformations and we have actions okay for all these transformation let's say if I asked spark that hey filter it will just create a block then I ask that hey do a union it will just create a block it will create this direct a cyclic graph in a very easy terms one transformation after the another it will just maintain in its memory that hey this is is what I have to do but it will not do that it is not going to do that okay and you can see one thing here because it doesn't do this uh we can go to actually I can show you the spark UI one minute can I show the spark UI how can I see this so if I go to Cluster single load cluster spark UI okay so we are going to discuss each and everything in depth there as well everyone so you will see few operations will be done there spark UI spark jobs uh do you see the join operation here everyone anyone of you is able to see the join operation here anyone able to see join operation here if I show you this description pay pay attention do you see join operation here so it says that till now it has done 19 jobs it has created right we see the jobs as well but anyone of you is able to see the join operation here has it done this join thing which I asked it so see I asked that hey do this joint do you see it happening here yes or no anyone will see happening it here yes vran so exactly it is the case what exactly happening in the background is that spark I told that hey I want to do this joint I want to do this join spark said okay uh have I used that join have I used that join somewhere can anyone tell me that if I have used this join anywhere as of now this orders customer DF have I used it as of now yes or no I have not used it anywhere right as you can see I have not used it anywhere right so what happened is that spark is going to maintain this Transformations so let's say it was join then I can have maybe filter and in a very simple or I would say in a very exact way this is what is known as d a direct a cyclic it will never maintain a cycle no it will always be one task after another and graph so this will never happen and the second you do an operation the second you do an sorry action it's going to show let me show you that now how it exactly will happen here pay attention everyone the second I ask Spar that hey orders. customer do show and let's say five okay see now now it is doing some spark jobs and it is able to show me if you go on this and if I just refresh it where is the refresh okay so it's refreshing see now when it is doing this DF it has done this join operation as well are you able to see this everyone now we see the join operation because when I asked it to show that is where it got to know that hey now I have to do some action and because of that action only what happened was that I now have to make sure that I do this join as well right everyone yes or no I hope this is clear this is in a very simple way spark lazy evaluation okay in a very simple way actually this is just spark lazy evaluation nothing else okay let me show you one more thing let me show you one more thing so let's say uh I do orders customer dot head okay and let me run this see I'm able to see this head okay let's say I do I run this head again let's say I run this head again okay so see uh okay I think it is not uh optimized just let me check so I want to show you the caching thing as well okay I'm going to show you the caching thing as well I'm not sure if this is by default optimized for that let me check storage data read uh has it cached it let me check stages and each and everything we are going to discuss here this is all the basics of spark so when I teach spark I make sure that I first teach the vanilla spark okay that is in which we will be doing your uh understanding of all these uis all these job stages storage all these things we will be understanding why am I why am I going to do that because uh many time what happens and rather majority of the time I have spent the most time on this particular page when I was doing my project on spark I had some jobs which are failing a lot here okay and to be a good data engineer again you have to make sure that you're able to understand things at this level right not just that you write a code here and everything is fine no uh what if something fails okay so just to show you one more thing if let's say I run it again if I'm just running it again let us see if it will create another jobs so See Spark jobs are created it is skipping it you will not see any time here but if I do it for tail if I do the same thing 48 so what I want to show you is basically we can we have some things known as cache as well we can actually cache this data frame as well okay so that it is stored and we don't have to calculate these things again and again if something fails now again this is all the property of spark rdd everything so don't worry on that but yes majorly this is something which I just want to give you an idea here also spark job was created okay then we are working on that and all these things is happening but yeah this is all the basics of spark which we also need to understand and uh basically I would say cover right everyone so now let us create orders underscore products uncore DF equals what we have to do here everyone so we have the order details data dot we can now join this we can now join this we can join this with our products data and what we have to join it everyone we have to join it on product ID and I have to do a inner join now again uh these are some basic syntaxes of spark no worries if you don't know this but yes should be I hope it is a lot easy to understand so what I have done now pay attention I have the order details product ID is here and the products is also having the product ID so I joined on these two things okay everyone I hope this is clear moving forward what we can do in a similar way and this is all the transformation which I'm doing as of now I just had the raw data as of now I was just having the raw data nothing else right everyone I hope you're able to understand this that if I go back into the architecture you will see one thing I got the raw data as of now into my Lake now I doing this data transformation which is my joining which is my filtering which is my all these any operation map operations all the operations I'm not doing here only right so I can now join order customers and order products on order ID so see if I join this now and if I just show you the final DF just let me show you now this is not doing anything as of now but when I will do final DF Dosh show it's an action and when I run this all my data is now basically here right everyone all my data is basically here which you can see as well anyone has any doubt on what we have done till here again I've done a very basic level of this thing you can do again more level of things you can get the insights all these things also we can get okay just to show you some code for that just to quickly show you some code for that uh what I'm going to do is uh let us have some empty cells let us have percentage MD then derive insides let us run this so MD is markdown and this is some code okay so what I'm doing SP Fire spark SQL functions columns sum and description and we can get the customers by total spending so we can get top customers we can get the bestselling products we can also get the order by country so you will see all these things see Germany is the uh highest like which is having the highest order these are the customers which having the highest oh sorry these are the products which are sold the most and these are my highest customer like the customer which orders the mode who spend most money so I can do all the transformation and other things here as well which is your uh any inside Gathering or anything I can do all of that I hope this is clear anyone has any doubt spark job is a series of task or one particular task uh kichu spark is a language or rather I would say framework which we use for working on our uh this thing to make sure that it is optimized enough spark Works in a way that it it basically implement or execute everything within after it gets an action okay so spark is not some series of task or something like that no spark is just a framework we are using which help us to do this distributed computing even if I had these multiple of these clusters with me I won't have to change anything in my code spark is going to handle the same from itend okay everyone so I hope this is clear anyone has any doubt here anyone any doubt anyone anything which is not clear anyone any doubt series of tasks that work together to complete a computation majorly yes but I would like to answer that spark is not that uh it's framework right so it's a proper architecture I can show you that spark architecture so let us keep our that clear as well its proper spark is like a proper technology which we installed on each and every so not like just programs or something no this is exactly the spark architecture where we have custom manager worker notes executors all these things are there again everyone uh pay attention so in this course I have spent and kept almost three four weeks for Sparks only because it is very important to understand this all of you can do the project still but what if Tomorrow there's a need for something else and believe me I have spent three months on a project where I was not able to optimize a spark job what wrong was happening there so that these are the things which makes you different from a very good data engineer than a normal data engineer who is not able to I would say who just know the things then to actually understand these things okay everyone so let me do it dark theme as well I think that is helpful yeah anyone any doubt what we have done till now so we have done this transformation you can see in front of you we read the data then we have work done on these transformation we also drove the insights as well something which we have not done as of now so in my raw data I am able to work on each and everything I hope you like this anyone has any doubt anything till now or are you good to move forward any doubt anyone please let me know in the chat if you have any doubt anything which is not clear we'll be more than happy to clear that please make sure that you understand this we have multiple components here so I've included those components so that your learning and understanding is a lot uh right toward is the right end so that you when you work you feel that confidence okay okay so anyone any doubt mkr what is this oh sorry I miss this most of these data transformation we can perform using Panda so what will we can additionally provide to client with data brakes uh vikrant see again that's the thing pandas works on a single data set what if your data set say let's say GB is in size so can you think of an use case let's say your uh orders data in Amazon or somewhere where this data is having GBS and pandas cannot do distributed computing I hope you understand that yes or no so when you have to do these things when you have to do these things in a very quick way for your streaming data or for a very heavy data then you can cannot use pandas there right that is where Spark comes into the picture it is very optimized it is very fast than pandas for especially for distributed computing pandas cannot do that right so that is the major thing how will you ask Panda that hey this is where the data is kept this is where other data is kept please go and work on this data there is not something which we can do I hope you I hope that is clear I hope that is clear everyone cool everyone so I hope we are good to move forward please let me know if you have any doubts I'm just seeing the chat anyone has any doubt the data stays just minute the here and this datab this is a lot optimized for working on distributed commutation as well as of now we just doing this basic things right but if you have to work on GBS OFA problem right uh hello everyone I hope you're able to hear me back uh what just uh like my full phone internet got off a just make sure that I get problem with this at the time of class only okay so let me just quickly see even running select start from table query can bring your production server down huge data set exactly V so that is a problem and uh one more thing so how many of you know that we can actually create tables here as well so just let okay let us use AI as well okay uh create a final DF to sorry create a final DF spark table and show how can I do analysis on that some example query so I like to use uh sorry test AI if it will be able to do what we want but let us see so pay attention everyone uh what we are able to do okay everyone just also confirm me I hope everything is fine like you're able to hear me and everything is fine the reason is uh I just changed the internet uh my phone internet got used up Okay cool so you see this everyone uh see I just sent that hey final DF I'm creating a temporary view of you so we will get a final DF table now see there are three ways you can work with spark rdd data frame and SQL and just understand one more thing now if you know just SQL using spark based on your SQL knowledge you can do distributed computing like that just opens a whole new array of I would say things which can be done using spark something which pandas and everything can never compete with so this query which I have written this will be distributed and run across cluster and I don't have to even learn a thing everything will be handled by Spar see this everyone yes or no I hope everyone is able to see this and these are the small small things which we have to understand if you have to become a good data engineer if you have to actually be a kickass data engineer all these things we will be doing see I've just created a view so anyone who is now comfortable with SQL write these queries they will work with spark spark optimization okay I hope that is clear everyone anyone has any doubt let me know quickly else we will just uh move forward we have some time so 10 minutes more I'm going to take but yes uh hopefully that is going to help you so meanwhile let me just go app uh registration so I think I forget this tiep uh that's why I was just thinking where why we are not able to find our keys app registration if anyone has any doubt let me know to solve problem does it depend on how the problem to look into rdd Data frame or SQL uh no kichu majorly we don't use rdd as of now in the course again I will be uh working on that I will be telling you what rdds are okay so those things are required but uh we can solve in any of the ways so we can work with rdds we can work with data frames we can work with SQL as well so I have worked with data frames as of now if you will see so I worked with data frames I what I would have done is I would have done like I would have just made these tables okay so let me just try okay let me just try if I can do all the things which we did uh 1 minute yeah so let me just try to get this code via AI only so it can quickly generate it uh make SQL table for all four data sets and get final DF which I did via data frame so I'm not sure how smart is it I normally don't use it in uh like when I try to write code but right now we can do it data frame approach okay so let's see if it can refer to a uh above chat as well above sorry code but see now it is creating views for this everyone okay now it is joining them so where is it yeah so let it right see so if anyone doesn't know data frame uh spark data frame and SQL there are a lot optimized rdd is now something which we don't use okay but still I will be teaching you on that okay so see it has created a final DF query order customer and DF it has uh worked okay this would have just required another uh code so like I'm sending all these four as well so this were the major four data sets right everyone what we started off with okay so I hope this is clear anyone has any doubt in this let me know in the chat quickly we have done lots of things here we have actually creating a production level uh product here sorry project here so I'm going to just register this now everyone pay attention P spark resource intensive in which way G so we have this running on a cluster right like what are you comparing it to exactly so see everyone these are the things which we will be requiring okay one minute so what I'm doing here is uh notes and let us have a new note how can I create a note here one minute everyone I have to paste [Music] it so let me save it somewhere just one second everyone so now what I'm trying to show you is so if you remember we have to create that thing right my laptop is little slow I think it's using lots of resources no worries so uh if no issues are there everyone then what I'm going to show you is that how can we create this connection so if you remember right just see if you remember this password or authorization case was left right everyone so for that we need few things so that is what I'm going to do or if uh you feel we can do it in the next uh session as as well which is going to be on Saturday let me know quickly if this much is clear then I think maybe we can do it continue it in the next class but yeah that is what we have to do now we have to now use synapse right we have to use synapse so that we are able to work on this just like in a SQL way we can just serve this data now so see we have now worked on this data and get all this data we have to serve on this data right so let us do that in the next session don't I don't think it's a problem we did a good level of spark uh work today which I think you all have understood we have seen the spark UI as well to got the idea okay so this maybe we will do in the next class don't worry on this uh so what we have to normally do I can just quickly tell you we have created this from here right now we have to we have these things uh again don't try to use it I will be deleting them as soon as it gets over we have to create a new secret as well from here and using that then I will be able to make a connection so I have to create a new client Secret I will just have to send so see what values it was asking if you remember everyone pay attention it was asking for application ID and we have this uh it was asking for application ID we have the application ID here then directory ID we will be getting and there is one Au okay secret ID which also we have to pass so yeah that are all things which we will be getting bya app registrations okay so I hope that is clear everyone anyone has any doubt uh one second I think it doubt no I don't think it should be required if it will be I will teach that I think it's procedure level SQL right so yeah should be fine I don't think any problem is there normal SQL should work honestly so yeah cool everyone so let us get back and yeah I think good choice to just uh we will just do this part should be fine in the next class we will do this part maybe on Saturday I will just show you how we can send it back and just use the Dos uh synapse as well today we did a very good part this data breaks right this working with spark and data breakes it's a very much in demand skill right now you must be seeing that okay why this is a very easy code and stuff but let me tell you we can do like n number of things here it is very much helpful for the today's data level the way it connects with your cluster the overall compute and do all these things and if you understand in depth about caching in spark okay about how the operations happens in spark about spark optimization different kinds of join in spark broadcast join salted join you will be actually having a very good knowledge and then on top of that if we do these kind of projects this is going to help us in enhancing our CV this is not a very straightforward or easy process believe me we have been using many Technologies okay so yeah that was all on let me just quickly go back everyone so this is the course which we are going to start from December 21st timings are 8 to 12 uh chis 10 is still valid so if anyone wants to enroll into the course please make sure that you use Chris 10 token okay if anyone has any doubt anyone any doubt is there please feel free to reach out to me okay I'm just sharing my uh credentials once again so how you can reach out to me one second so if anything is not clear please Fe me to ask the same we'll be more than happy to answer that and meanwhile let me just share the links with you all so you can easily Reach Out so you can reach out to me either on LinkedIn or please feel free to connect to me connect with me on YouTube we'll be more than happy again this is where you can check out the course uh the same link is there on the bottom which is there there is a number as well so this is the counseling team if you have any doubts just make sure that you reach out to them they will make sure that they help you answer that uh will not be pushing you towards the course that is the thing if there is any other course which they feel or maybe the udmi one the simple 400 one they will just ask you for that as well if you are a fresher or something so yeah do that ramakrishna yes you can pay in US dollar I think it's a lot competitive pricing we have very less PP like pay parity applied I think it's some $120 or something in us so yeah I think that's a pretty reasonable amount considering in us okay so you can pay in US dollars or even in UK currency as well but US dollars I think will be better for you because it's U not that expensive even in us okay cool so anyone has any doubt uh before we stop just let me remove this screen and yeah cool everyone so just tell me quickly if you have any doubts anything which is not clear and we'll be more than happy to do that now I think we can have generic doubt as well so if anyone has any doubt in general with respect to Big Data or career in general I can just take it and we will just close the session in the next five minutes okay again just to go forward this is from where you can download the whole curriculum as well uh snowflake is not there okay does the code work uh ramakrishna just try that out once I think it will work just try it out uh not really sure from the back end if it doesn't work just let us know and we will see that how we can make it okay hello I have a final year project on an app to identify plant disease what could you advise me uh I think initially is a very easy project to do like you can just go to see the plant disease for example you can use deep learning Frameworks here right uh depends on what see the thing is first let us think of is from an objective perspective what are you getting into the data is it a images is it some data like the iris data set just apply the basics yeah I think it's the easy project to do now okay reach out to me on YouTube or Linkin and I will help you but yes the basic idea is that only uh prerequisite gamer9 9797 is this which we have which we said that we are already going to provide you with the recordings so python SQL and database Basics these are the prerequisits okay I already enrolled the course I have a good ETL development experience on ssis how can I use this course to enhance my profile I have three years of experience in total uh praque we will be starting with the course uh if you have a good experience in ETL in general uh please make sure that you are a lot attentive when I explain spark because before that lots of things are going to be Theory again that is also required but in spark is V where you will be understanding a lots of things apart from that when we do projects please make sure that you understand each and every step of the project and then what I will suggest you is that if you are in the field already try to see the projects which you have done already can they be done using those Technologies can they be enhanced by using those approaches or not okay I am a fresher pursuing MSC AIML this course with for me with no experience uh tun see the thing is again that if you are in AIML field and you are in Masters right first please get the basic programming experience if you have that then this can be helpful to you because companies do look forward to these technology IES and especially from Master's guy if you don't have any idea behind coding or whatsoever then first please get that that will be helpful okay the requirement are take a picture identify the disease provide subst sustainable solution to treat the disease to the user uh hini see the very first thing which now what you can do is very easy now you can just directly use J or any any uh framework there as well so how I will do the project is I will first try to see if it is limited to some particular plant then get all the images and everything work on that create a de planning solution along with that maybe you can now to directly hit your apis the chat GPT apas and stuff and get Knowledge from that okay they are also as good as it can be and then for the overall solution sustainable solution I think they should be good to give you if you want to just do the project if you want to do create a deep learning model then get the data again the data is going to be a lot useful there right so get the data and you should be good there so basic idea just to draw for you quickly is going to be something like this two parts we have first get different images like get multiple images create a deep learning model second don't do anything use chat GPD or some API use hugging phase for the POC to get the proof of concept based on this once you get the diseases right or wrong sustainable solution uh either now try to create a database which tells that okay what can be the solution because this is very vague as of now like what exactly is the disease what uh kind of plant you are looking forward to but yes get the overall approach here get everything in a single prompt and it should be good uh I have just joined a startup how to practice the mlops project shivkumar uh if you have joined the startup you will get lots of Technologies sorry lots of opportunities if you are working on mlops first understand ml that is very important thing go to any Cloud try to make those end to end projects that is what you will help you in mlops directly nothing major honesty uh use sagemaker use uh take up a server and try to make and deploy your model there see how you can make it cicd like how you can use uh your GitHub actions airflow all these things for continuous uh create these pipelines that should be more than helpful if you know ml then just do projects and you will be good in mlops as well that's the straightforward easy advice my journey in mlops has been majorly been I would say supplemented by me working on a research projects and projects in companies that is all what I done honestly for mlops and I was able to grasp everything once you grasp the basics now then I don't think it's the difficult thing I know Python and SQL no experience in sense of job experience will this be suitable for me and duration of the course uh duration of this course will be 5 to 7 months uh because we are going to cover things in depth okay if you have uh Python and SQL and no experience in job I think it can be good for you just you have to make up your mind because companies will be looking for these skills in your CV okay so yeah that is going to be for sure be helpful Pro also provided you are already in the I think you were the person right you were in AIML yeah you are already in a IML so data will be forming a big part of any job which you are going to do so having data engineering knowledge surely is going to be help hi I'm currently working with more than two years and I have wrote python script to retri data from list into an PR Oracle okay can it be considered a data engineering project uh not really RP that is just part of your project so I think this is just this F first part you're just doing this data inje part using this thing uh not a data engineering project see the whole aim this is a full end to end data engineering project because we have got the data we are transforming it by using it's torn down but it's a full NN data engineering project we will be making this synapse we can make a serving layer here as well using that we can just then ask anyone that hey you can get the data so this is a full kind of a project now I can throw in uh the scafa and other things here spark streaming Kafka to make it little bit more uh I would say complex so this is a full idea or I would say chain of a very of what you will be doing honestly if you ask Mina 60 70% of projects will be into this pipeline only just that it will getting get a little bit more and more complex you might have to get data where are different sources data might not be that straightforward so this as your data break code it will be a lot big you might have to spend days on those code okay it will be a lot heavy in size so maybe you have to optimize your joins as well so these are the things which you will be working on majorly okay uh in my I major work as a python developer but by current comp randomly assign me project like currently they have assigned a project related to J uh TP I think that's a learn take it as a learning experience honestly uh that's how you learn so take it as a learning experience if you getting these projects unless they are just totally out of your domain like if you are in the data field and everything if you know Python and stuff uh that should be good okay isan next is your sir I have a skill set of ml DL J ml Ops okay in depth as a fresher doing with super good project what to do now what package can I expect isan uh totally depends on lots of things uh if compan is coming I see you have not done DSA if they have the first round on DSA you might not clear it plus when you say that you have all these knowledge I might have to take an interview and test you on that that exactly how good that knowledge is is it just bookage knowledge have you actually created something to enhance your paycheck please create something which is very much useful that is going to be more helpful because just having the knowledge is not going to help that I can be sure of okay DSA intermediate do it in advance uh yeah so not selling it or anything but I have a course on DSA on udmi as well everyone so if anyone has any doubt any problem or want to get started in Python and DSs you can start with that it maybe costing you 400 500 cool then yeah uh I think the doubts are clear it's already 10 so let me just take one last one I'm confused okay is there any equivalent to of as your HD insights AWS yeah yeah shortly we'll be there let us see as your HD insights AWS Amazon EMR HD insites Amazon EMR we have now Google we have data proc this is the most basic thing sorry I get confused because there are many services so prati we have EMR now for that though no worries I think in that data proc on Google EMR here cloud-based services that can process large amount of data now see straight forward okay so I hope now let's just let me close the outs see the idea is that what separates you just to give an example I completed my whole journey of mldl everything myself from different blogs uh diff andng courses and stuff but the major thing which helped was my internship and research projects which I was able to crack oo rooms then crack Goldman Sachs along with that a very high level of DSA skills DSA system design to many people have studied that what else you have done that now becomes a differentiator plus how good you have studied that that is also one very big differentiator uh in my conversation you will always find that I understand the problem from a very basic perspective and then build on top of that many students they just get confused in different different uh Tech techologies jargons like if I ask person that hey this is what I have to do it's a simple thing right data injection okay if I don't even know Z data Factory I will go and learn about this many student they will be like no I have never done a z data Factory I have a problem in this I don't think I can do this so what that is the biggest separator the problem solving mindset so make projects data created on the internet are always better Henley okay uh no worries thanks a lot everyone hope you like the session vikrant if you are in data engineering field please learn datax datax is a lot in demand okay also tell me I will create a proper in-depth video I'm actually also already working on this like two months back I started so I have the content ready and I just have to shoot it and I've started actually shooting it as well but data brakes so shortly will be useful data Brak is getting used a lot in the industry right now okay kol I think the doubts are clear I'm also a little tired so graph DP I have taught that as well uh okay so just last 2 minutes uh which application words if you have application words then tell me that if feedback is there then tell me that I'm tired neck is paining a little so we will close the session okay uh for the doubts please reach out to me then or in the next session you can ask that K may you are confused h and please everyone just make sure that you check out the course uh everything we have make sure that we have worked on this a lot me and Chris sir okay no one else majorly is involved I also talked to many people in the professional like in the industry please if you are planning to look into this just consider this and if you have any doubts please free to reach out to me uh VII what we have done to was not at all easy believe me it is something which will give you I I hope it has giving you a very good idea that this part right to get the data into the data Lake and then work on that so you have actually transformed your draw data into something useful not that easy thing to uh sorry do or uh like I would say start with okay so spend the time see this video again face out reach out to me okay content related out reach out to this number or just reach out to me or Cher and we will help me to answer that a doubt I think who was confused one minute h h h okay what do you have to do after python uh V you will see the recorded lectures this recorded you will see on Chris sir on the live channel uh for PR recorded for sorry for prerequisites we will give give you the recorded things on this uh once you enroll right you will be able to login on this krishn Academy learn. krishn Academy there you will see the recorded sessions okay doubt just I'm clearing that I'm seeing can you please provide me with a context uh ke you have done python but uh like right now are you a student or what exactly that will help me to answer that your is the last out I'm taking okay everyone so yeah everyone meanwhile for all others thanks a lot please consider the course consider reaching out to me subscribing to my YouTube channel and Sir as well also if you have any doubts come to me uh any feedback any appreciation please let me know in the chat that is also going to help a lot and we are going to meet on Saturday okay okay I'm a little tired backgr I cannot answer in a V terms I want to make sure advice so the advice which I give that is a little helpful just please give me a background okay that is going to be helpful last out I'm taking yours only so it's clear kg gamer if you're in art and you want to come to it sector first just uh Learn Python from any channel okay you can even go to my channel there's a hello python series see if in one month you can spend that time then you can just come to me again sir I spend the time in learning python I know that okay coding is I have a little idea up tell me if I should do this or not and in that you will find a oops video as well that even technical people are not able to understand very nicely that's around 5 hours of video see that once if things make sense then come back to me then I will tell you if you should what you should do for IT field road map just do coding first if it is for you or not okay cool H let me just share the link of course once again it is uh basically number this is the number and then this is the link okay so I hope I'm able to sync my hands and the link but I hope you get the idea uh okay so I hope you not getting this uh cool [Music] so thanks a lot everyone H I was waiting for your doubt or your little context so after python please if you student try to do either on development or towards the data science side I will personally prefer data science side because that is directly in relation with python and AI so which is a lot in I would say demand today and yeah that will be all without context that is what I can answer okay so cool everyone uh bye-bye I will be closing the bridge we are going to meet on Saturday complete this project and I will see that what I can give you a little bit more idea about in the next class it can be Hado it can be spark internals of spark internals of aop so yeah uh simple video will single video will not be able to help you that okay but I can make series of