Live-Implementing End To End Big Data Engineering Project With Cloud

Key Takeaways

[Music] [Music] hello everyone uh I hope you're able to hear me can I just get a quick confirmation in the chat so welcome to the day three or the session three which we are doing right hia hi Shas let me just share my screen as well screen screen one and yes great so today everyone the agenda is going to be with regards to the project so I'm going to show you and we will keep it a lot uh slow but yeah our aim is going to be that we are able to understand what exactly are there in Big Data or data engineering projects so that is something which we are going to do today okay everyone uh it will be done on Azor so in the last class we saw jcv and in the course as well so the course which we are bringing to you all on which starts from December 21st which I will be coming to now in a time uh we will be using uh all these clouds so that you have an idea right great so before I begin and let other people join for a couple of more minutes like it's 82 as of now can if someone has any question from LinkedIn or YouTube you can just ask that meanwhile let me just showcase to you so this is the course which we are talking about let me send that in the so you will find that in the description as well everyone uh second it will be a little difficult but yes you should be able to follow that should not be a problem so everyone this is the course which we are talking about complete Big Data boot camp with AWS and Azor okay let's see if the chis 10 is still valid because I think there was a limit and yes so you have 10% still off right please feel free to connect to this number the counseling team if you have any doubts with regard to the course if you have any Q&A just tell me and I will just answer that these are all the details for the course so big data with Azure and AWS Cloud uh we will also be using gcp so that's all the clouds you can see which we uh majorly I would say see in any job Mentor will be mayang Agarwal that is me and Krish Nayak that is Krish sir start date is December 21st timings are 8 a.m. to 12:00 p.m. IST on weekends Saturday and Sunday and proficiency level is majorly professionals because data engineering or Big Data that's a w case or W I would say A Course in itself or V text tag in itself which is used a lot if you are a fresher and you want to still apply make sure that you have credit cards and stuff plus you have the prerequisite which I will be coming to in a bit clear and other information you can just learn about the course here the cabus you can also download from here I have shared the link everyone okay so yeah coming to the common Q&A yes this course will be from scratch there will be prerequisits of python and uh your databases the videos will be provided for that so we will be providing recorded videos for this so you don't have to worry on those but yes there are prerequisits okay coming next uh let me now take up some questions before we start so first 10 minutes normally we will keep two questions and then we are going to go to agenda so just will come to myself as well and this is going to be our agenda for today I will try if we can complete the project today as only if uh like let's say it is not everyone is able to follow up then one two and and a half hours it will take for this end to endend project to complete if not we can continue the same on Monday as well right but I want to make sure that I give you a gist or the proper idea that how this is going to be okay great so one minute you can download this from here everyone I hope my voice and everything is fine uh voice video and all okay my bad extremely sorry I think I was not sharing my screen uh so this is the overall everything about the course which I was mentioning so you can see uh each and everything which I said so Chris 10 is still valid you can go to this particular number as well you can download the cabus from here each and everything is mentioned so these are the prerequisites which we will be providing okay everything will be from complete scratch okay uh Maya see when we talk about beginners there are two cases one beginner is someone who is in college okay let's say who don't have any idea about coding uh if you are that level of beginner then I will surely say that it is not for you I will very sincerely suggest that first you should get a hold of coding and everything second are the one who are let's say started their career okay let's say they are in first or like see we have written it very nicely right here that it is majorly for professionals but still students or many people are coming who want to understand if let's say they are working somewhere they have four five years of experience but they are beginner into this field they are let's say from data analytics field or from non Tech so those are the one which I'm saying okay cool let me take up some questions quickly no worries I have N9 years of experience as full cheack developer I want to move from data engineering uh sure sayi this will surely be helpful so if you will learn from this course that is going to help you a translate into the data engineering part okay B all the things which you have learned in fullstack you will get a little bit idea that how the data is flowing and all that also will be a lot clear right uh development or data engineering uh apurva it is not that straightforward I just Mark the questions which I'm taking so that everyone can see this uh Aura see it depends on the kind of problem you are solving uh in the development as well it can be pretty straightforward and it can be visce versa like data engineering you are just making normal pipelines so the kind of problems which you are solving that will actually Define where logic is getting used okay so it it is not very straightforward in black and gray that this uses logic this doesn't I have done development as well and data engineer project as well in my case data engineering projects were the main which took up a lot of my effort lot of my uh I would say mind space that what exactly has to be done okay development maybe I found it easy so it was straightforward as I've done the same in college as well whereas data engineering or these uh data whilea part I just kind of got the exposure in my uh job only but yes it will depend okay uh Mahindra Vu uses Azor for data engineering use cases like we will be using today those are majorly said as Azor data engineer there is no difference uh there are lots of these job titles okay aor data engineer uh gcp data engineer AWS data engineer it's just that you have to first understand the basics and everything right with respect to data engineer and then when you use these tools okay it's the same as Mac User Windows user Linux user I hope in the basic is that everyone is using using a computer and we are just making sure the different os's we are using and in this course as I said each and everything will be covered from complete scratch all the clouds today I'm going to show you for the Azure as well so that you get an idea on that okay uh Mohammad if you don't have any experience with uh Azor that should be fine and a as well uh Today class again everyone so as I hope you understand that uh I'm just trying to uh make sure that you do this project many things might not be clear but I will try to explain you that in the course we will be spending easily a week or two to understand each and every uh This Cloud infrastructure right so EX on Azure I will be telling you each and every of those things if someone feels that that can be done within an hour or within an two hour I'm extremely sorry that cannot be done if anyone is saying that to you uh you can take it but believe me it will not be clear we will be properly spending three to four hours to understand each and every of these clouds before we work on them okay I'm great getting stucked in fine tuning BD English modern uh Sardar U can help you with that but you have to reach out to me separately so this is not related to this course unfortunately I can't get it how can you cover both clouds in eight month they are so vast uh rajit see uh I'm not saying that the V okay there is a difference again everything which will be used for data engineering right that will be properly covered and that can easily be covered it is not that vast honestly we will be using different Technologies and you will get an idea that exactly what happens okay and I will be majorly focusing on the basics first so that those Cloud you will just see as a tool uh it's the same as uh like let's say maths right it is yes it is so vast and everything but for machine learning someone can just take a month and explain you all the maths so we don't have to do rather no one can do proper Mastery in the cloud or in anything okay you cannot Master AWS I think within ears as well but things which we have to use for big data use cases that is something which we will be focusing and we can easily do that within eight months so yeah recorded video will be available uh Vin yes the course will be from scratch uh again just one thing if you are someone who has never done coding please either wait for your Demi course or first do the coding and get your Basics clear uh we are not taking in people who are let's say complete who don't have any experience with coding okay great slus I have shared again that is all fine prere requested so I have shared again my I hope your doubt is clear as well anik I from India Delhi so somewh from Pakistan uh yes kashan you have to just reach out to I think the number once and you can of course enroll from this website Okay one minute uh I hope rajat your doubt is clear then I am data scientist with 11 years of experience so which things of data engine shall be help me as a data scientist ano see uh if you're a data scientist with 11 years of engineering uh majorly you will be moving now towards the way where you will be managing now data science projects right the heart of data science lies in the dat data and how you manage or work on that data and something which every data science majorly do you have to work on that engineering or the pipelines part of that as well right something which I will show you today as well like exactly how they are getting made and why it is very important to clear but yes all these things we will be you will get an exposure to so on that note normally data engineering is a good field for you again it's totally up to you on where you want to make your career but data engineering or Big Data technologies will be helpful to you so you can take that call easily okay great uh that should be fine uh D see if your Basics and everything are a lot clear then I think you can independently just check out those uh GCS and aure aure Cloud okay I think you will be able to make a lot of sense with it majorly the things are same only like let's say we have data Lake in all these clouds Google has uh your cloud storage then Azor has ADLs Amazon has S3 so if your Basics are clear in that way uh I will hon suggest that first try to First figure out yourself if not and if you think that your Basics are also not clear then I will suggest you to take the course because if you're already working pretty sure many things will be clear to you okay great so yeah let us now move forward uh we need to take subscription uh so Community we will be taking the trial today as well I will be taking the trial so yeah okay cool uh just to Now quickly I think 10 minutes have passed uh 2 minutes overrun uh now let us get back back to the proper cabus funds okay so this is going to be the agenda everyone cabus walkthrough plus Q&A so that I do for the first 10 15 minutes so this big data boot camp with AWS and aor and of course gcp as well is something which we will be focusing on as you all can see uh this I think I have covered a lot of time in the last two session as well this is the third session today everyone so this comprehensive program is equipped to skills and knowledge all these as a big data engineer we have these all three clouds each and every everything will be taught to you from Basics okay and we will also be practicing on the clouds and doing lots of Capstone projects which are used okay today Project should give you a little bit of idea how exactly we will be doing it one minute yeah so one more thing everyone I hope you're able to see me in 1080P today uh yeah I think yes it is fine great so let us move forward then everyone uh these are the prerequisits python SQL and database we will be giving you the recordings you don't have to worry on that the recordings will be provided for that if enough students faces issue after seeing the recording then honestly I will be happy to just conduct extra class as well for that and we can cover the same okay let's move forward then everyone we will again start with big data Concepts the overview I'm going to just quickly skim through them so that you have an idea that what all we will be covering because in the last two sessions as well which you can find on the live you will be able to see them okay now before I move forward anyone has any doubt anything which is not clear like with respect to the course or anything like not the Q&A which I have been answering regarding to you I will come back to them I hope you are able to follow each and everything all my voice video and the screen sharing is all good anyone has any doubt in that so let me know I can see that today like we selected the HD so it's in 1080P and I hope you will be able to see the what all is written pretty clearly because we have to code a lot today as well okay yes you can see the last two videos that should be all good no worries on that great so let me just also show you guys one more thing so just to show you like uh for the intro video which I took on the first session you will find the video on my channel as well everyone okay so I have uploaded the video for the introduction to Big Data what exactly is Big Data I can share this link with you all of you so that without in the so this was majorly in the first session where we walked through and this was discussed so let me go here and you can watch this and for the previous video which we have seen already you can go to Chris S Channel okay this is live already and along with that you can go to live and these two are the previous sessions which we have took everyone okay I hope that is clear great so that should just I think clear all the doubts and yes let us move forward cool uh great so that present Cloud which is the best plot and how can we find which the best okay let me come back to questions anyway everyone so we will start with Hadoop uh architecture ecosystem everything in depth again hdfs architecture then Yan then map reduce all of these will be discussed then we will see and learn about spark right we know that now map reduce is not used that much but we will still write code in like I will just show you an example of map reduce then you will understand that okay what exactly is the problem we will also discuss about rdds then moving forward we will learn about data frames and data processing then we can move forward with Advanced Data processing and optimization in spark so this is actually something which we have to understand in a lot depth the reason is that we use this a lot so in spark optimization actually follows or is used a lot this is something which I have spent months on on optimizing the project so that it can be done and I was able to actually uh I would say reduce the time of one of the operation from six hours to around 45 minutes based on these optimization and you can think that okay how better or how good that is right then spark performance op optimization and tuning then moving forward we have no SQL databases and again no SQL databases will be done from scratch we have the mongod DB then cassendra is there then we have the hive architecture again the data store for uh which is inbuilt in Hadoop we will be using that then Advan High features then moving forward we have the streaming now so Kafka will be there everyone then we will also do Kafka producer consumer everything in Kafka then SP spark structure streaming we will see pubsub and Amazon all these we will see so that you can have an idea again when I will be teaching you let's say about spark streaming and Kafka I will tell you that okay what they are based on so that tomorrow if you're using any of let's say if you're using Google pupsa it should not be a problem for you okay great uh then we will start with a par airflow for managing and orchestration of pipelines data pipelines and then we will start with cloud computing in the cloud again a your will be uh basically sh done uh today I will just give you a kind of an idea that okay how we we will be using these things then aure data brakes we will be using after that aure data factory data orchestration then Advanced Data Factory transformation monitoring everything and basically EMR we will start with AWS AWS EMR is same just like we did Google data proc in the previous session awsr is same then AWS S3 and then Athena and glue for serverless quering and then we will start with projects so we will be doing minimum five projects and actually I plan to include more projects so that based on the how the class is going we will have multiple projects which you can use and even uh actually I would say uh work on that add that to your CV and explain them in an interview and pretty sure that if you explain that properly interviewer will be more than impressed because these projects will be in a way that someone who have a year of or year and a half of experience those are the level of projects which we are planning in this particular course okay cool everyone so if anyone has any doubt uh till now let me know with respect to the uh the cabus and the session anything is there let me know else we are going to start with today's agenda that is the Azure account and project understanding I kind of skimmed through it the reason is that I have already explained that in the last two sessions okay each and everything you have present here most of common doubts you will have here so if everything is clear just give me a thumbs up or yes and then I think we should be good to continue and one more thing everyone so this Chris 10 is still valid uh it is valid for I think first 100 uh some customers so please make sure that you are uh quickly uh major uh just registering for the course if you want to because we have limited seats in this we want to make sure that the experience is pretty good so we are not taking uh like let's say thousands of students here we have a limit here so that we can provide a good experience to each and every student so if everything is clear uh if anyone has any doubt just ask me that in the chat if not then I think yeah we should be good to move forward okay uh SBI I have shared the pr requests a lot of time so I will suggest to just see from here uh see that uh in the course as well it is written Okay cool so let us now go back and yeah project will be done again in the course uh no Sakina so I will again start this is kind of an easy project if you ask me right we will be doing little bit Advanced project this project I just want to give you a hint or the taste that okay how the projects are in data Engineering in the course we will be doing little Advanced project okay uh willo YouTube course will be taking time uh sorry Udi course will be taking time AWS V I think you can just see on the website many services are free and everything so we will be working on free trial so yeah okay Mahindra classes will be on weekend morning 8 to 12 okay morning 8 a.m. to 12:00 p.m. you will be given the recordings the dashboard each and everything will be there for you you will be getting the community support as well I will be answering the questions all these things will be there if you just download this the project list is also there so Capt project 2 3 each and everything is written in the course uh have spent a lot of time in writing that and the first class I also walked through it you will see that these projects okay you will see one more thing that we have mentioned and seen that okay which Technologies we will be using so it's not that I will be making you do the project a single project with four different data sets no we will be using different different Technologies here we are using Kafka and streaming and a no SQL database in this project we will be using data factory data braks all these things are the ones which we will be working on we will be using different data for matths as well okay so yeah that is what we will be doing in projects what all of you can do please go here and just download this and go through it we have spent a good enough time on making this okay I hope this is clear everyone if anyone has any doubts still just let me know and if not then we should be good to move forward so great I think doubts will come meanwhile let me just move forward duration will be 5 to 7 months uh Mahindra it's a fully comprehensive course it's not an intro course honestly to Big Data it's a full course which is going to cover each and everything okay uh Sakina we have actually set up the time from lots of uh consideration with people who are from abroad as well uh recordings will be provided so I am pretty sure that it won't be a problem you will have the recordings you will have the community you can reach out to to me for the doubts and yeah those all will be there so I don't think you will have that issue okay uh sa as I said so we will we are not having this course as this is kind of a professional course okay not an INT level course we might have seats full and I will not be taking uh basically registration then the reason is that we want to have in this batch we want to make sure that we are teaching in such way that everyone is able to transform their career and with that in mind we we will be limiting the scope and everything okay great so this is the screen of today's agenda everyone I think the cabus work through and Q&A is done now I will be working on these two things so the first one is setting up as your account and the second is Project understanding let us first understand the project and then we will be uh like moving forward with the your thing okay great so let me now explain you all the projects everyone okay cool so first let me explain you the project and we will see if we can do the project my main aim will be that I'm able to explain each and every one of you properly not that I have to just skim through and complete the project no I will keep it in layman terms many things will be there okay uh I'm just writing it in red here many technical part right now might be unclear because again in the course we will be spending time like for example I have a proper dedicated class to exp explain you as your in four to 6 hours right uh each and everything is your but again in the YouTube we have to just make sure that we are doing the projects right so to give you an idea if anyone feels that okay something is not clear just tell me in the chat and I will try to make sure that it is easy I will explain you in an easy way okay so yeah great so let's understand the flow of project okay the very first thing is uh it is going to be in a way ETL based offer project which we use in Big Data engineering so we will have some e-commerce data e-commerce data and this flow is actually independent of any cloud or anything right so don't get confused in anything this flow you can do in any of the tool okay so we will have let's say customers we will have orders we will have projects right so all these things are going to be here everyone then what we have to do once we we have this project or sorry once we have this data we will be ingesting this data via uh timing Z I have already explained okay so as your data Factory so as your data Factory okay so I will explain each and everything don't worry we will be using multiple as your services so you get a taste that okay how exactly things are done and I will try to keep it a lot easy but this project is a good project like to understand how the flow happens in company this is a pretty good project okay great once we have done this we will then load our project into so a your 1 minute as your data Lake okay so it is also known as a LS Gen 2 so see I will tell you the major mind behind this Pro uh this project uh in a company there are multiple sources from which the data can come right for example in a e-commerce way let us take the example of Amazon right in Amazon the data can be of different types and it can reside at different places okay so for example let's say uh Amazon uh someone went to Amazon via Instagram so you see those ads right then people are also using Amazon app people are using Amazon website okay so all these things are there through which data will be at different places I hope each and every one of you agree that data will be on different places that should not be a problem so the very first thing when I say the e-commerce data it can be on different places I hope that is fine Eco coming why is Eco coming suddenly everyone is the is there Eco in my voice can anyone just confirm that quickly okay Del that but should be fine yeah uh Prat can you please check at your end once uh sorry janardan you can you please check at your end okay that will be helpful uh great okay so please don't uh disturb the flow like these technical issues I have to make sure that I stop and then move forward anyways so first let us discuss around the data okay so we have different different data sources right let me just clear and make it again so that each and everyone is able to follow so I will explain the exact flow as well so that you are clear okay everyone so the very first thing we can have different different data sources okay so this is actually also data source we can have data normally okay let's say in our local normally that is not the case but we are still going to see this we can have data stored on see we can have data which is coming from an API let's say you have written some API we can have some HTTP where the data is kept and we will be using this https today again because we we want to understand the basic flow is this thing clear everyone that data can be stored at multiple places for example for Amazon there can be data which is there can be orders which are coming from app which are coming from Instagram like when you click on Instagram and then basically someone comes to Amazon from there so you can ers coming from there you have the data sorry orders which are coming in from the application then from the websites so we can have different different data sources okay so yeah next once this is clear we have to make sure that we use something known as aure data Factory and and see these are the pipelines which as a data engineer you will also be creating so because the data is at different different location okay so let's say for an example if I ask you that hey uh how many orders were done yesterday you will have to make sure that there is some Central repository or some central place where you are able to get all the data so that you can tell me that okay how many orders were done yesterday right so orders from the uh application order from the website order from let's say some direct party some business orders yes or no everyone I hope this thing is also clear that we can have orders which are coming in from different different locations now if I ask you that hey what were the total orders don't you think you will have to collate all the information at a central place yes or no how many of you agree with this thing right now this data Factory okay this data Factory will help you to un to basically connect to those data sources and then dump or sync your data somewhere okay so after this we will be using something which is known as data Lake a your data Lake ADLs okay A D LS Gen 2 where we will store all the data so each and every of your data will be stored here right so this is a data Lake now again what is this data Lake and everything discussed in the previous session or normally in the course we will be discussing that how data L is kind of a central repository where all the r is present okay it is stored in the block form so someone is asking for S3 just like we have hdfs the way we store data in Hadoop S3 Sakina you can just think that we are storing the data in some uh I would say block format okay in S3 which has a which has its own format for distributed data storing right so in Hado I explained in the last class that we do storage which is distributed okay and similarly we do processing which is also distributed so Amazon follows its own kind of this distributed storage which is S3 I hope that helps to understand that Sakina in a very easy way okay so everyone I hope you able to understand this data L as well ADLs Gen 2 anyone has any doubt in this thing any doubt anyone great once we have this data once we have this data stored then what we are going to do we are going to work on this data so if let's say someone is asking that hey uh what were the orders by country what let's say how much did people from USA ordered what was the amount for this we are going to use Azor data Bricks now we'll be taking a separate dedicated session on data braks and Spark where I will be showing you few things about spark in this week only but for the timing you can just think that you can code on that data okay just like you have some data stored you can code on that data and majorly spark is used here and again when I will be teaching it in the course you will have a hours of dedicated uh work on I would say data brakes a your data brakes we will understand each and everything there spark to we will be spending around two to three weeks where we actually understand each and everything in spark okay once we do this once we process our data so this is actually the process part process part we will be storing it back on ADLs Gen 2 so we have kind of this input data we send it back after processing back to data Lake and then there is a service which is aure synapse which we will be using to connect to this data and then understand or make SQL queries and visualization I hope you are able to see that how this is not a very easy project just that we are doing lots of things and this is a proper use case like in many companies or rather in my uh experience as well I have done project in a similar way where we were getting some data then we were storing it into Data Lake then we were working on the data doing lots of processing on that data finally storing the results and then making some uh writing queries or let's say making some visualization on that so with that I hope the V flow is clear anyone has any doubt in this let me know if anyone has any doubt uh Prat I will show you that how we can get the data don't worry on that okay so we will be seeing that data we can create partition so when we are storing the data we can create partition based on something that's a little complex but yes that can be done and when we will be reading data via aor data Factory it can keep on syncing the data depending on how you have basically configured it it it can keep on learning or reading the data based on that okay and yes the partitioning can happen that should not be a problem we can create folders and stuff based on the ear and everything that should not be a problem okay that's a little complex and advanced stuff but yes we do do that so should be fine great so if all that is clear everyone then let us just see about Azure once so Microsoft Azure okay this is about aure exactly the way we do it okay now what if you remember in the last video in the last class what I explain you is that we go towards the distributed computing structure right so I explained you that how we have multiple PCS which are connected either you can make it yourself or what you can do is you can use some provided by this uh overall uh any of the cloud service provider we saw gcp last time we created a server uh it was having one master node and two workers node in a similar way Azor also have lots of things which it provides so we can just get start with Azor then we can go try Azor for free so see these are all the things available only to new your customers free monthly amount of 20 plus popular Services okay then we have USD 200 credits this is what we will be majorly using everyone so just to Showcase you this is what we will be using and we will be using a credit card for that another reason we are specifically like that we want professionals or people who are in job so that they will not have any problem in having the credit card or debit card to uh basically register for this now once you go on try for free 1 minute let it run if anyone has any doubt till let me know I'm not sure why it is working so slow but yeah meanwhile I will explain you a little bit about aor as well so let it run and here we can learn a little bit about aor that how things happen in aor yeah so this seems good so it is going to ask you to sign up uh one second let it load okay so it's actually asking you to sign up uh you can either sign up from uh the account so you will have to create a new account then uh let's say if I was just want to sign up from here okay so let me just enter my password quickly 1 minute I hope this is the one if not then I forget my password but yeah okay so password is wrong just one minute so everyone you can just see uh so you will have to create an account of Microsoft account to use your as your for sure okay uh no not stay signed in let it do authorization so again uh it can just do that what meanwhile what I can do is uh it is authorizing and everything we can go as your on this browser as well where it's logged in so this is just to Showcase you this process again will be explained in the course properly it should not be an issue okay started with this your sign in cool so actually this is where I've logged in already so I have started this because it was going to take time like normally when you make a thing so see it is asking you identity verification by card you have to agree to some Services you have to fill a few things very important thing is here as well so it's saying that no automatic charges will be here after your credit card is over we will ask if you want to create with pay as you go if you do you will only pay if you use more than the free amount of services the main thing to note here everyone is that no automatic charges will be here okay so I think many students have this doubt I hope that is clear now right I hope that is clear everyone if anyone has any doubt just let me know till now uh it is going to give you 13,300 of credit for the first 30 days and once you loged in it's going to use you are going to see something like this it is also a free account everyone so as you can see uh I'm getting a notification 16,700 credit remaining right so we have these credits which are remaining and it's a free account only okay is it possible to work with Gen on Big Data uh yes Prat like depends on what kind of project you are trying to make okay like again gen or the data like you are using the data right so you have to handle that let's say if you're trying to work on J and you're creating models which is going to work on data so how the data is reaching to you that is all what data Engineers have to do right that is all the work of Big Data engineer so I hope that is clear everyone anyone has any doubt till now this is going you will going to see this again I'm going to explain you few things here right so let us go back see when you create an account on aor UC see a management group right in that you can have subscriptions so for example I created an account and I have a single subscription it's free subscription as of now as we have seen and I'm gotten some credit 16,700 I think okay then we can create a resource Group so a resource Group is like where you can mult group multiple resources okay so I'm going to show you each and everything don't worry on that we can group multiple resources in a resource Group I hope that is clear anyone has any doubt in that anyone any doubt till now any doubt anyone and then we can use multiple resources so all these Azor data Factory ADLs a your data brakes this will be coming inside my Resource Group okay so let us now begin uh this how to create an account again you can see and I will be making a video as well that should be fine but for the timing let us move forward let us first create a resource Group okay so Resource Group I have few of them created I'm creating another one if anyone has any doubt let me know uh AWS again uh Community we will be trying to use the free tier only should be fine so you can see it's free trial okay I am taking here southeast Asia so this region and everything we will also be discussing in depth don't worry but as of now you can just think that I showed you multiple data centers right so I'm using this region and let's say it is e-commerce life okay next let's go forward we don't have to do anything here we can just review and create let me see yeah so it is saying that validation has passed and I'm creating a resource Group Resource Group has multiple resources inside it so each and everything which we will be doing now it will be inside this e-commerce life project I hope that is clear everyone anyone has any doubt in this let me know anyone has any doubt till now for everyone uh I will just quickly explain the project again that what exactly are we going to do okay so one minute one minute everyone great actually let me do one thing uh just a second uh I'm trying to explain you the project in a better way so maybe I can use one note here so if anyone has any doubt in law let me know if anything is not clear so just to show you uh one minute let me create one new file and let me try to explain the project here as well so that each and every one of you are able to understand it what exactly are we going to do okay so in that actually not able to add images properly let me try here if I can do that yes so so pay attention everyone uh just going to explain it once again that what exactly are we going to do okay so that it is a lot clear see we will have some data source right as I explained you we can have multiple of these data sources that should not be a problem so we have these multiple data sources first we are going to read from use a your data Factory and something which I'm I'm going to show you just as we move forward what what I can actually also do parall I can show you these things so let me just get rid of few things many windows are here but yeah okay so see data Factory okay so this is the your data Factory I'm going to create a new one Resource Group we created e-commerce life let us use that then region we have to use southeast Asia name we can do e-commerce live only okay version V2 you don't have to do anything here next configure get later so I will just explain quickly everyone subscription there is free trial okay so see we will come back to where are the data sources I will show you that as well don't worry but first now let us see about data inje so data Factory we will be using for that I'm creating a new data Factory free trial Resource Group is the one which we created earlier so I'm using that Resource Group only I'm giving a name e-commerce life region is southeast Asia version is V2 we can configure get later that is not a problem networking we don't have to do anything here everything is good you don't have to touch anything here tags and anything we don't need and we can just go directly to review and create so it's doing some validation okay and we can create click create now and it is going to create so you will see deployment in progress so I'm creating a new resource which is a ADF a your data Factory resource inside my data inside my Resource Group now I can go to Resource and you will see see that e-commerce live is there you can see data Factory V2 and we can go and launch Studio here so aor data Factory Studio okay now I'm going to show you how we connect to different different sources here a your data Factory as you can see okay so here we have data source and we are trying to use from data integration via data Factory so let us go back once and here now if you will see we have multiple things order Monitor to Monitor all your jobs okay we can manage to see okay what are the different Link services we have the Learning Center again as I said I'm just going to skim through things in this class but normally in the course everything will be taught from complete scratch so don't have to worry we can actually create pipelines here if you will see so what I can do is I can create a new pipeline okay so let me create a new pipeline I'm going to get rid of this thing so closing this and closing this as well here I can just do Ecommerce eCommerce pipeline okay everyone so I'm doing e-commerce Pipeline and I can just remove it now here there are multiple things which you can do so the one which I'm going to use is copy data okay so pay attention just I'm now trying to create a pipeline now these are the kind of pipeline you will be creating in your jobs as well okay we have multiple things here synapse a your data Explorer we can have data bricks here as well so just like in Microsoft Services many of you will be aware that in word we can have an Excel table so in a similar way all all these Services they are kind of interl or coupled with each other inside here only you can actually have data braks as well you can have synapse as well but we are going to use them separate so we are able to touch multiple or maximum of services okay now here if you will go you will see Source here okay so for example this is the data Factory this is my data source I have to tell where my source is right if it want to reach the read the data I have to tell where the source is in this we have to create a new one and now I think it will it is going to make a lot of sense Amazon S3 Amazon RDS okay Apache Impala blob storage you can see multiple sources here everyone I hope you're able to see that cassendra couch base db2 okay we will have FTP Google ads file system hbase Google Cloud Storage hdfs HTTP Hive so I hope each and every one of you are now able to see that we have multiple storage here mongodb my SQL so isn't uh this the case that our data can be anywhere so all I'm asking is that this says that hey okay tell me one minute yeah from where I have to read data that is source and you can see that in data Factory how many things are there how many things are there from which you can read the data so is this clear each and every one of you that we have multiple sources just like I said that your data can be in a SQL server in a no SQL server on Amazon S3 and you can see in front of you that a your data Factory as a service is supporting multiple of these things you can have these databases file GIC protocol and many other things as well no SQL I hope that helps you everyone so if you have let's say a no SQL DB at your end I can read the data from here is this thing clear or not at the end of the day everything is data I can just in a way mention that hey read the data from here and then I can work on that that is not going to be a problem okay are these different storage to get data yes Sakina so these are different different uh I would say places where you can have your data if you are getting confused if you are not able to understand it you can just simply see that there is MySQL and there is mongodb as well so it can be a overall no SQL or a SQL DB as well everyone is this thing clear yes or no anyone has any doubt in this and I hope you can appreciate that how we have multiple of these things from where we can read the data many of the services how they are coupled with each other so I have the Amazon S3 here as well I have the Google Cloud Storage here as well see GOOG Google ads Google big query Google Cloud Storage so all of them are interl with one another and that's the benefit even in the clouds there not that if you're using Azure you cannot read data from S3 that is not the case if you want to read the data from S3 you will have to make sure that you are able to just provide with the credentials so that Azor can go to your S3 and read that data is it clear everyone yes or no anyone has any doubt till here anyone has any doubt yes if you're new in field of ml engineer reach out to me maybe I can help you with the refil okay so recently one of the company actually reached out to me and they were looking for hiring so yeah fine great so let us move forward now what I'm going to do now github.com Myan 953 okay we are going to read data from here but how to actually store the data in S3 or gcp so Sak I will be telling you uh in a similar way we will be creating Amazon uh sorry AWS lake so let us now move forward everyone I hope the source part is clear the source part is clear okay now let us see that how can we store the data in somewhere okay so now comes the role of data L Gen 2 which we are going to create so pay attention everyone this service it can read the data right we have seen that it can read the data from multiple sources and let me show you that it is going to read the data let me just use an example uh this is the Big Data project so I have created customer CSV data okay I can just show you how the data is and this is there for everyone so each and any one of you can refer to and see this data okay uh s I'm going to explain that here as well don't worry on that okay you can read the data from here what I'm going to do is I'm going to go on Raw so that I can see the CSV data here okay so you can see this data everyone this is the customer Data customer ID name email and Country right this is the ID this is the customer name this is the customer email and this is the country okay it's a kind of a dummy data but I hope you can understand from this that what exactly is going to happen now what we going to do I'm going to select HTTP here so this is a HTTP data as you can see this is a transfer protocol like some Network thing but yeah I hope all of you are good with that now I can see that this is a delimited text this is a CSV comma separated value comma is a delimeter here so let's select delimed text and continue now we can create a new linked Services okay so see HTTP server what I'm going to do customer data here instead okay we can have the authentication Anonymous we don't have any authentication and I can push the base URL here at the end of the day isn't my aim just to make sure that I'm able to read the data everyone yes or no once I do this no authentication nothing is there we can have multiple parameters as well we can have this Advanced thing as well we can add some Dynamic content in Json again these are little Advanced my aim is to make sure that you understand the flow for that I can do create I am not importing any schema first row is the header so if I go here first row is the header right everyone and we can just do okay now and just to preview the data I can see this preview data and let's see if it is able to read the data and I think it's able to read the data yes or no everyone so no matter where my data is kept using aor data Factory I can read this data is this thing clear each and every one of you I hope you are able to to understand few things here that I have got my data from anywhere in the world like this GitHub anywhere can access it right and I showed you multiple sources as well are we able to read this data yes or no are we able to read this data no in a similar way tomorrow if let's say Amazon it has data somewhere in the SQL DB somewhere in the no SQL DB can't I read and get all the orders together yes or no everyone are you able to understand this are you able to understand the power of this Cloud as your data Factory that how this block this block majorly is able to read each and everything I hope everyone is able to appreciate that now this was about reading the next part is something which Sakina was asking that hey now how can we add data into this data Lake which is your S3 or anywhere else right so as just as of now I have read the data now what if I have to store that data somewhere as well okay so let let us go back and see the next service before I move forward anyone has any doubt till now in the project what I have done till now anyone any doubt please ask that in the comments again this is going to be a in a way your proper uh job level project we are going to create multiple things here as well now okay so I'm going to copy and read other data then connect these pipelines so something which you will be also doing in your job as well right everyone if anyone has any doubt let me know okay and this is in a way you are creating an inje pipeline I hope everyone is able to connect with me here that creating these inje pipelines everything this sound like a very difficult task but here I hope everyone is able to appreciate that how in a way how easy is it right so all I did was minute all all I did was I have a data source within data inje data Factory I'm able to see all these things just that I have done right uh Mahindra has showed you that you can actually connect to lots of the sources right so Mainframe data what exactly do you mean by Mainframe data you have that so you can create a new one here and you can see all these things here and you can then have generic protocols each and everything is present here so I don't think anything will be there which you cannot find here