Elevate Coding Malayalam Starting a Career in Data Science Faculty: Dinesh Karthik Raveendran

good evening everyone hey yes good evening so myself Dinesh Karthik and in this session we are going to look into like how to uh start your career in data science this is what this session is about and like so let's get started first like uh sure so yeah I'm I'm known who is going to talk about like how here you're going to start your career into uh data science so who am I so I have to tell who am I to start this uh your career in data science I'm basically a data engineer who has almost a decade of experience working in different uh companies different organizations starting from small startups to bigger organizations uh in different uh countries so that's what basically I have that's my uh career background I mean I can go deep dive into it later if needed so yeah let's start it so who am I targeting in this uh particular session the targeted audience are basically like undergraduate under our postgraduate students and it freshers who are looking for new entry into it or who are Juniors having like a year or two of experience and trying to figure out like which is their like uh which is their calling like which technology is going to work for them or what they are passionate about and also like any data Enthusiast whoever wants to work with data just to understand what is in the field of data so that they can make some decisions later so those are the target audience so the level of uh explanations or the information that are put in this particular presentation is for those audiences yes so uh how uh we have how this particular session is uh kind of schedule is like we will have like four uh Parts the first part is like looking into like what so generally discuss like what is data science and like what are the usage where it is used in your day-to-day life and all those things will be like we will be discussing in the first part then the next second is why like okay yeah I you are telling me to start like it's a good idea to start a career in data science why is it a good idea we'll be looking into this like what are the trends and why this can be like uh why you should like start a career in data science especially yeah and the next is like what is like a roadmap like how we can from a student or like someone who is like not from the data background like how you can start your career into like different ones and also when we are talking about the why we need to carry in data science we will also be looking into like the different or like the commonly uh available opportunities generally data science I mean we can discuss into generally data science does not only means data scientists there are a lot of other uh other roles which are actually uh kind of closely needed inside the data landscape which go hand in hand and we will discuss about it and what are the roles and what are their responsibilities and what are the commonly uh skill sets and technologies that are required and are used then the how again like so it's like a common uh Road plan or like a journey in which you can follow on like reset gold it's like a simple one we can also explain on this and finally I will reserve some time for us to ask some a q a like anyone of you can put your question in the chat and during the Q a I can look into them and answer them so that's what we are going to focus on this today and how this is the agenda so yeah the first one is like uh what is data science uh simply like that's my word I will put like it's art of collecting data and making sense out of it it's not simply you can say hey there is a data and like this is what it's never the case uh the like you get a lot of data and like the data will never it's like I mean ideally you'll think like okay I get data I can find something maybe that's like when we are looking in some tutorial and we are studying that's the case but when we go into the real life implementations or like in work scenarios or professional scenarios you have data but it's not straightforward to make some decision out of it and make a business decision out of it so that's why it's a big process and that is a huge industry around it and always the data will also be not like as easy as it as it is so yeah so what is data science so generally like yeah I would like I put right uh data science is an interdisciplinary field it goes from end to one starting from collecting data properly storing the data making it retrievable and then looking into the data finding some patterns coming up analyzing it coming up with some uh kind of like predictions or like proper insights everything revolves around data science I mean generally when someone talks like uh data science people are like okay yeah I'm a data scientist no it's like everything is kind of like considered as a data science is an interdisciplinary field so that's why you have to look into like what are the various different uh analogies or like different roles and responsibilities that are available you have to find some which will be uh suitable for you on which you would like to work on and then go for it so that's basically uh take higher level of like how I would say uh data Sciences so like yeah so generally I said that's a simple explanation right yeah okay I collect data I analyze data then then I use the data and make some information out of it ends it out of it so generally like okay so you are saying that some it is there so where do I uh come across it in my day-to-day life that's a major question so first like I put like a basic applications like where you are actually using an end product of data science which you don't you might not even know or you already know the very first thing is like internet search when you go to Google Yahoo Bing or any search engine data science plays a huge part there like how what result is shown for your uh search query and like what about the search term you put everyone won't get the same answer that you are getting or the same search results the reason is like it is custom tailored for you based on the data that is available about you and also the different ranking mechanisms on everything that goes behind will give you like a very curated list of results that the machine believes is going to help you and most of the case it works with a better machine learning algorithms and everything and secondly I would say is like recommendation engines which is also like day to day it comes with example like you're watching here in YouTube live live right in the right side you can see a lot of recommendations and suggestions which is like very custom titled for you and most of the time it's something that is going to interest you and you will always watch it similarly you take anything like Netflix obviously you get very curated list of what you are going to like Instagram Tick Tock like how they become famous in the sense like you're kind of getting reels are like I don't know in Tick Tock like though I don't know what the terms they call there everywhere you get the same videos that is going to keep you hooked like this is something that you are going to like this is up to your liking so how does the system know like it's not only two persons who is going to view the application so like I know if there is going to only two or three person going to view this I can say hey this is a video for you this is a video for you but we have like millions or billions of users accessing it at the same time and the system has to give them the same should not actually it should not give them the same image or like the same video it has to be coming up with some very custom trailer for each and every user to their liking so that's a huge implication starting from collecting data properly storing data with big data Technologies and then writing ml prediction algorithms deep learning neural drugs or whatever and then coming up with some basic or like your recommendation and showing it to you hey this is something that you are going to like yes then finally and the other thing is like day to day you use it like image or voice recognitions right like example if again if you take Facebook or Instagram any effort you upload a picture it will automatically suggest who is this person and do you want to talk just tag this person in this post that you uploaded this image that you uploaded how does it know so again that is also there and like voice recognition like most of us have might have used like uh what uh Google home like Siri Alexa everything so we just tell something it's like it understands converts The Voice whatever it records into like binary like it's vectors and everything then it kind of makes sense out of it come up with like a proper a response to it uh take an action based on what you provide it has to decide whether it has to give you a replay and ask for a further questions or it has to go for the next level is like okay this is that it has a definitive action this has to be taken so should I go for it so all these things will be part of it so that's also used day to day and the other is like fraud detection mostly like as uh especially in India we have like uh online payment they become like the go-to it's like that becomes a norm it's not like most of us carry cash it's like we go everywhere we pay with the card or like UPI scan a QR code pay it yeah we are done that's how it's going and also there are a lot of scams going around so how do we prevent that one of that is like fraud directions so data science field because they have to be very methodical and how to find a proper way because it's like huge amount of money is involved so they are the ones who initially come up with fraud detection and everything now it's kind of integrated in everywhere so they know a pattern of like what are the purchases where are the purchases you do and if it's something entirely out of the ordinary they will just immediately block it and like most I'm not sure if any of you have got it if you are doing some purchase which you have never done it is entirely out of ordinary for you uh most of the bank won't even open the payment you will get a call immediately from one of the bank representatives to confirm hey is it you I mean not for small amount but for very huge amounts you will get all these kinds of things happening so these are something that you do like you are interacting with data science it's become a part of a day to day life and we are not actually aware of which or most of the people are not aware of it like what goes behind the scene and the next is like this is I would say like a power user or in the sense like Advanced users who are like a little bit more tech savvy how they are kind of like use day to uh technology is the end of today example so that I just took it very focused for like uh software Engineers or like tech people here in this case uh obviously everyone might have like heard about chat gbt there's kind of the real fuss like I mean obviously it should be and it's a conversation a uh to be honest at the end of the slide I would have put like uh I use chat GPT to create this presentation in the sense like whatever the things I put I asked chat GB to proofread it it looking for grammatical errors spelling mistakes and is there any of a like is my using NLP like I already had fed it with my own data like how do I usually write my post and Linkedin are my blogs so it already know the how my tone and style I write if there is anything out of the ordinary or if I have to uh stick with my old uh a stone and style it'll actually suggest me and say hey this is how you write it use this or if I have like saying that hey uh this is like the entire thing that I want to write about but I want to put it more professional I'm not able to find the accurate word for this like then it'll give me the suggestion for what I should use so most content creation is now kind of like Get most content creators and who wants to write a lot of things or basically using chatgivity I mean it's going to evolve it's going to become better and better and better in the future so who knows it may even be doing like your homeworks at some day but hopefully not that no one is going to learn properly yeah then the other thing is called GitHub co-pilots I'm not sure how many of you are aware of GitHub so GitHub co-pilot is a new offering similar to chat GPT chat GPT is a conversation a whereas copilot is going to be like you have this plug-in Integrity into your idea where you write your code in vs code sublime or like pycharm or you can actually have like code editor their workspace and co-reter in GitHub you and integrate this plugin you kind of write it so it's basically like how two of you sit together and write a program that's bad programming similarly the co-pilot will like whatever you you write a simple context and say hey this is a function that I'm going to write it it kind of goes through it it was already trained with data from all the open source code that are available in GitHub and it it is going to like suggest like a standardized or like a formatted more better code like simple skeletons or anything so that you don't have to write the skeleton entirely from there or like commonly reoccurring functions you don't have to write them again it will automatically like kind of like give a suggestion and autofill it you decide if you want to use it or remove it later so these are like the day-to-day activities of like for normal people as well as tech people that's how it is kind of going uh like your data science activities yes so yeah so we just have a initial presentation of like what is data science how it is used where do we interact with it in our day-to-day life so yeah then the next is like you can ask okay that's fine so why do I need to have a career in data science that's the major questions everyone is like okay everyone asks like okay what is the benefit or is it really a good field for me to work on so simply put a tagline like I said like when you run a big business you cannot simply say hey I have a feeling if I launch this product or if I add this future this is going to make us more money it's not going to work anywhere nowadays every company wants to see some data behind it and make sure like based on this data they can make a proper product development or any decision they are going to make they need to look into their proper figures some proper numbers so obviously data is going to be used and indirectly yeah so everyone who is going to work around data are needed so that's kind of a gist of like why do you need to start a career in data science if I go further so just to give you a high level of why as you can see as of 2023 like beginning of 2023 we have so by end of 2022 we have 94 Zeda bytes of data globally available across the world in our Digital Universe so you can ask like as you can see right I just put a simple math or simply it's like one setup height is equal to 1 billion terabyte so you can see the scale of data that we are talking about that is currently available across the world and out of this 94 Zero by it more than half of it are like generated in the last just last two years like kovid and post covered so Kobe kind of triggered this everyone talks about it right it kind of triggered the digitalization like it kind of kick-started or pushed the visualization into like five years ahead so now everyone is kind of using like work kind of most of the companies become remote or like even hybrid or like everything's people started using a lot and lot of internet and people are using technology and obviously it ends up producing more and more and more data so that's why you can see this 94 database can go grow like exponentially in the coming years so if you have that much data we need people to work on this data like again I said the flow collect those data there are this much data available collect those data properly organize them store them make it available for retrieval for others who are going to work on the other section of it then people who has to uh retrieve this data make some simple analysis or like come up with hypothesis for like prediction machine learning build models deploy them make it available that's how it is and also following that app so these are the other simple things that I kind of came up like I can also I have put like where I actually got those numbers it's not just I purely made it out of I'm not just putting those number out of 10 year as a data person I also did my research to come up with these numbers and they are actually published numbers in a particular kind of Articles or like proper sources I have put them at the end of this presentation as well so here uh as you can see by 2022 the annual revenue of like big data so like all the data industry is 56 uh above 56 million just in a year just for big data and like it kind of grows around it's and also this is mostly uh focused on us a little bit of global this number is I don't have the exact number that's why I put like above 56 million that's a minimum number I'm talking about so then the next one is like uh also there are other there are some research and it is being done and after all the data Trends big data and a trend growing on 92.2 percent of the big businesses are already claiming that they already invested investing or already have invested in big data and AI that's what they are currently doing and the next like okay this is like business-wise money-wise we look but a general normally if you take like okay is this number going to keep on increasing so generally if you see per second an average human or average person who has access to a phone laptop or whatever is producing a minimum of or an average of 1.7 megabyte of data per second including you and me even in this like we are producing more and more data like I'm streaming data across platforms and it's being stored somewhere and being retired by you and like someone like in also behind the scene there is a huge analytic stuff going on how many people are watching what are the comments going on people how do people reply to this command do they like it or not or is there any abusive comments or is there anything that doesn't other to uh YouTube's uh Community standards then automatically there is something even before those command gets posted they will kind of get kind of deleted or if someone puts in a busy comments or something which is not meant for it like a Spam or anything then someone will kind of report this obviously like good citizens are like people who are going to net CNS will kind of report it hey this is some scam like people kind of report scam or report spam whatever in this comment and obviously there is again automated pipelines that are going to look into it uh filtered categorize it remove it or not make decision okay this is not something that cannot there is not enough uh positivity or confidence that is this is an abuse or anything like a Spam so I need a human intervention the machine decides it and takes it to the next level uh which is like some human look into it remove it and all this so even like in this web I'm speaking and new listening here a lot of things goes behind the scene yes so then we talked right why all this is happening and now I'm going to say like as we said like there's Sciences like International Fields there are a lot of things going around and here I'm going to measure I'm only taking like uh some of the very major or like a bigger umbrellas or like bigger vertical Tower you call it I'm only talking about those kind of career tracks that's kind of like hugely available there are a lot of niches and everything going around and also here whatever I'm saying right take it with a grain of salt in the sense these roles and responsibilities are like uh how to use it they are very fluid in the sense depending on organization to organization some of them may be like interconnected there can be a place where like a data analyst and data the data scientist does the same data work as a data analyst or like that time we pray is where there is no animation learning engineer it's like a data scientist who have qualified enough so they are working on proper software engineering or like a data engineer working a data analyst what is them doing or it also be like uh like I said I put like data comma ba analyst in smaller companies they kind of like work together on the sense the same role the same person does both of it when it goes to very bigger companies they have like a very uh definitive section like okay a ba analyst versus data analyst so all of things are going to be kind of like very specific to companies they can change a little bit here and there so why I'm doing what I did here is like kind of put like the very Global or like the very commonly used stuff that's what I'm kind of putting it here so uh the first is like data or the being analysts usually their work is like they analyze the data which are kind of already available and they make sure like they come up with some they find some patterns or insights come up with them and it is being used for making this in the decision or to understand how the business is working on the next is kind of the data scientist the data scientists are more of like similarly they are consumer of data but what they do is like they are more mathematical or statistical Focus they kind of come up with hypothesis trying to understand the business try to find some predictive ways or using statistical analysis or statistical method that they based on their knowledge they kind of come up with some hypothesis try to prove the hypothesis with the POC and all those stuff then when uh the business decides okay this is actually valid and it is being like bringing result and everything then the ml Engineers or the machine learning engineer comes in so what they do is like they are like properly like hardcore software Engineers who are purely focus on like predictive analysis and those kind predictive analysis or predictive models those kind of stuff so the hypothesis the algorithm whatever that is being generated by the data scientists like they write some code and they come up with something which is like a privacy kind of a pace like a prototype so these machine learning Engineers they take it and then they build it as a scale like we are talking about here right how do you flag a command if it's abusive okay there are simple a data scientist can take some data set write it into your local machine or some with a very small kind of data set they kind of run a predictive model and yeah they can say hey this is how it works so it's fine it really works good then the data the the same data model or the predicate model is being converted into a proper uh machine learning model then it is being deployed as a software application because like it has to scale for billions of people per second like even now if you take YouTube you don't know like how many millions of billions of comments are kind of like through fed or like they are checked for abusive comments or all these things so they will be done by Machine learning engineers so data Engineers so these are the guys who obviously me so uh we are the folks we kind of sit make sure that the data is uh we kind of looking to make sure the data is available uh taking like how do you put it I kind of like retrieve data like I said like you said right you put a post a command it goes to the application but for for this data to be available for the consumers or the other sector of the spectrum like uh data analysts data scientists machine learning Engineers uh the data Engineers kind of make sure they build design and build like proper infrastructures and systems which read this data or collect the unstructured data available everywhere make it like a proper structure and store it in efficiency and make sure they can be retrieved in even efficient ways so that the data analyst or VA analyst data scientists ml Engineers they all can use it so and technically all this yeah even though they are like different career tracks within data science umbrella they go hand in hand one cannot exist with others without the data engineer collecting any data and storing them the ba the other three folks they cannot do anything because they need data there it's not their expertise to kind of like work with the big data collect them and do this and a data analyst obviously can like I'm just like a data engineer cannot just put the data and expect some uh the business to mutation out of it that's where like the data analysts or data scientist comes in analyzes data come up with all those stuff undo it and also the data scientists also simply cannot say like okay I read I wrote this hypothesis and it is working I have this working basic model yeah just the application has to use it it's not going to happen the ml engineer has to come in convert it and deploy it for scale similarly the machine learning engineer can I mean there are also good machine learning Engineers they kind of do the hypothesis but mostly it's kind of data scientist who does the hypothesis come up with all this with the business understanding and the Machine learning engineer display in scale and like I say said it is fluid there can be places where these two roles are interconnected data scientists and machine and engineer there won't be like a strict line between them but they may be doing the same or yes or no kind of stuff yes so now let's deep ever like I said like these are the top level let's go like one into another in this case like see like okay let's take a data and be analyst so generally they are responsible for acting and I mean processing dating statistical analysis like I said you have the data they kind of do proper analysis on the data process them maybe check for some stuff and build some uh dashboards for the uh like very simple right you are running a business you need to know how many sales you made yesterday or you're performing good how like do you have enough stock do you don't have enough stock all those kinds of things for the management or the business to run the data analyst and the B analyst kind of build all the uh dashboards or scripts or queries whatever they need to do will be done and these can be used by the management to make like proper data driven decision making like I said it cannot that management cannot say hey I think this season like this product is going to work they are not going they cannot sell something like this they need to have proper definition of like what is going what is going well what is not going well how is their business performing as a whole all these things they need so generally this is like a role skill set and a language required like I said the role is like the collect perform data analysis and make regular reports available for management so that they can uh it can either be like simple automated reports that sent via email or like proper uh graphs or dashboards build on any other visualization tools so that management or like anyone can go in the company can go log in and see that's what eventually a data and ba analyst is for this the skill set they required is database querying obviously they need SQL to query the data from wherever it is stored and they need to know like any of the socialization tools like how they have to visualize those data and in the sense like you have to be very sure in the sense like I example let's say I have a data search and that can be like n number of kpis kpr like key performance indicators are like points which I can come out of it I can build like a dashboard with like 100 or 200 of such things showing so and so numbers or whatever I can do that but like does it really going to help become business to kind of use all this 200 300 500 numbers or like for each level like what is it required for like a upper management what is required for someone who is working on the field what is required for a software engineer who's working on this product or what is required for a sales person who is working on this product so they have to come up with all these different sets and identify which is going to help which department and is it really like the proper number and what are they lacking and how can they fix those so those kind of things they have to identify and build so obviously they should understand the domain they are this company is working on their business model everything so they have to be a little bit more business knowledge to do all this so the commonly I have put like multi if you can see I have put multiple tools in the sense they are not everything is used uh so these are very commonly used depending on like company to company project to project you either they use one or another so that's how I put like the very commonly used ones so that you can decide which one is going to make sense for you similarly SQL SQL is kind of the basics of data the sense like any role you take you need to know a SQL without SQL you cannot do anything with data yes so first thing like I will go to the journey but now itself I can say with confidence you should learn SQL to start your career it doesn't matter any role if you want to have a career in data start learning SQL know itself yes then like yeah obviously like we said to visualize different stuff we have Tableau power bi Etc there are different virtualization tools you can start focusing on one of the tools become master of it and based on and also like the knowledge is kind of easily transposed from one tool to another because like the knowledge is again the business knowledge which one you need to show those are the knowledgeable and the tool knowledge is can be easily transferred from one tool to another so focus on one tool learn it now then later you can transport like based on what the company is needed you can change to another tool or like learn like different tools yeah the services and like depending on company some places you might you need to use like python RR to do some reporting stuff not like just visualization you have some number you have to come up with some certain numbers build some PDF or whatever or like take some analysis so python certain places they use Python certain places they use r or certain places they just go with pure SQL or obviously not or uh is or not this R programming language or they use like the Excel or spreadsheet obviously that's also how it started people initially put they gathered all the numbers or whatever they have put it into Excel sheet they just write formulas there select cells and that's how it started now like we are doing it in advanced way of like doing it in the database coming with big data Technologies and all this but still it is also needed certain places you don't be needing all this a simple xlc it will be enough for data or B analysts to come up with all this so that's why that is also commonly used or people obviously know when they are in this particular kind of uh career yeah the next is data test and specifically I'm not going in any order I just put them in certain order there is no order saying that one is above the other or one is below the other based on order it's like I'm just put there is no order in which I'm talking about these roles yeah so like I said uh data analysts work they are they are like very much similar to data analysts they are also very much uh business oriented they need to understand the business and then they mostly do like uh they did analysts do something called descriptive or exploratory data analysis lessons they kind of like explore the existing data or describe the data and come up with this kpf like what is altered present what is happening that's what the Emoji really look into Data scientists like they said they do predictive analysis they build predictive models for analyzing this data finding some patterns and see if this going to pattern is look into so it's like looking into the future like predicting okay is this going to happen or this is what this person is going to like in the sense like a recommendation So based on your past the sense whatever you the videos you watch in YouTube it is predicting the future like hey this is something you might like or you may like so similarly uh data scientists are looking into the future like how to predict something is it going to happen or what is the confidence or like the level it is going to happen those things this data centers majorly work so what are the uh excuse me ah yeah yes so um like I said the role is again the same they kind of discover patterns and Trends propose some algorithm strategies for the business and they kind of try to improve the business and come up with the solutions the skill set like they said it's very much mathematically oriented to be a data scientist you should be knowing Advanced maths and statistics because you are going to crunch numbers with the statistics and come up with some predictive models yes so there is a majorly required and like uh predictive modeling and there is a concept called a b testing so data that's one of the thing of product development they use a lot of these predictive and statistical models they run a b testing in the sense like let's say a little bit of like what a b testing is like a big company is not going to say hey in an overnight they cannot decide and say like okay I'm going to change this button from this color to this color or from this position to this position or change a feature just like that what they do is like they take a very subset of users and they try two or three usually a b two but most of the times like different as well they kind of uh roll the roll over the future for different users in different like different subset of users and they see like what is a success rate or they use it efficiently using it or they are not using it are they like not liking it they kind of come up with all of this like that's called a b testing so that it allows them to decide like okay whatever the hypothesis they made and the predictive that they are doing is it working in real life that's how they use a b testing to finalize all this and obviously our data analyst as well as data scientists are storytellers in the sense like okay you have all this data another until you cannot say or make a tell the story out of this is what is happening with either visualization or like simple presentations it's not going to help so that's also a skill set that is needed and Big Data Technologies in the sense like they don't need to kind of like know end to end in a sense let's say mostly they are going to query some data from like a big data lake or something so they should know like how do they should query because like coding a data lake or like a kind of a distributed system uh is going to be different from coding a simple relation databases like MySQL or like postgres or msql server so they should know the difference a little bit here so it will help them yeah as I said the tools and Technology they use are SQL databases or nosql databases and like different programming languages python or Scala Java based on like the companies what is required but mostly Python and R are like kind of the go-to for data scientists somewhere they use Java and Scala are basically some data engineer or like a software engineer who kind of transition who already have this knowledge they kind of still use it but mostly data scientists the top two language go to language or Python and R most of the cases and like I said Hadoop spark and how they just it's not like they need to develop the data lake or write the end-to-end kind of set up all those infrastructures or all this in those big data it's more like they simply know how to retrieve data like query those data from there efficiently and so that they can run their analysis and predict you uh thinking and statistical models that's what they need and similarly uh tensorflow pythons or these kind of like Frameworks that are meant for predict remodeling so they kind of run this and kind of use it yeah mixes uh machine learning Engineers like I said they are responsible for developing the algorithms and software that enables them to deploy the uh machine learning kind of stuff like these most of this model or hypotheses that are built are proposed by data scientists and they will be like converting it into proper software application a machine learning application and deploy it the role is like yeah they have to design and develop machine learning or deep learning I mean I didn't put like differentiation generally like anything that's go with AAA apps International Learning deep learning neural networks everything comes under this so I'm not saying that as a separate for the uh deep learning engineer or like a neural network Advanced learning Engineers nothing like that just put them all of them come into this category that's what I meant by Machine learning engineers and they kind of like perform the statistic analysis that are proposed and they try to fine tune it because whatever data scientists they propose or they build are done for like a minimum minimalistic use case in the sense like they have like very specific data and they do it like not in a real-time kind of a production environment they do it like in their staging or like a local Mission with a very curated data set they try it and all this but the same cannot you cannot directly uh take it and put it into the production environment and expect it to work so this machine learning engineer also fine-tune it like saying how efficient it can be made so they will look into it and obviously for this the skill set they need is to know about like Data Systems how do you build like uh data application interfaces a little bit of data warehousing like what is the efficient way is the data you stored so they can retrieve it and obviously they should know software engineering because they are like building proper software applications end of the day everything is software application you're building so those are the skills that it's needed and language used or basically like python Scala Java and R as is more of like an uh productionized applications it can be R is not widely used but it's morally python Scholars Java or depending on the language these are the commonly used languages and similarly they need to work with Hadoop spark And Hive and like here the cloud data Technologies are needed are not needed for data as analysts scientists because the sense they have access to a database or something they retrieve the data run some analysis but whereas here a machine learning engineer should know about Cloud Technologies like AWS Amazon web services or Google Cloud platform or Ms Azure uh all this so that they have to deploy applications they should know what are the available Services there or like available infrastructure there so that they can efficiently deploy their application scale them in like for millions or billions of requests per second so they have to know these Technologies and similarly like uh tensorflow pytarch spark machine learning kiras are the Frameworks that are basically used by Machine learning Engineers to build predictive models or machine learning models in scale and deploy it for a productionized use case and next let's go to the uh next set so obviously the last one or like in the sense in the order it's the last one uh data Engineers so data Engineers like I said are the how first part of this entire data problem or a data project that's how you can put it it's like they are responsible for like designing and building and also maintaining like the infrastructure proper data pipelines and everything to make sure the data is kind of collected no data is missed and the data is of proper quality there is no like data that is like wrong or anything that is being stored in the final data warehouse or data lake or data lake house they have different terminologies and different setups and there are different architectures so Lambda architecture or Kappa architecture they have different architecture for data engineering pipelines and everything so they have to process this store it in a proper structured way so that it can be analyzed by others so that's what data Engineers do as a whole so like they roll they have to develop test maintain the pipelines data warehouse and look into all the data architecture so data is not lost so you should never have a data the data should not be lost that's kind of a big problem right and also like you have to have like uh there are like different concepts like real-time data and batch data like how fast the data has to be reflected in the end system for the consumers is it going to be like if I click if the user end user click something now is it like data analysts or analysis how to have it in the next couple of seconds or microseconds or is it okay for them to have the data after and half or a day so those kind of things so they are called like the real time or not real-time pipelines and also like I said data quality is a big emphasis here so is the data up to the actual quality or is it not they have to look into it then the next one is like the skill set which is required for this it's like obviously database they have to everyone has to know the database or this data Technologies but here as a data engineer they have to know even on level like even deeper level in a sense let's say there is like kind of a data warehouse like solution distributed data warehouse solution let's say uh example Amazon redshift or Google bigquery the the end user they just need to know how they can query it efficiently but for these users they should know how to store them efficiently so they should be deep diving into those Technologies understand how the index works there how data can be partitioned separated uh sorted and stored properly so that they can easily retrieve without costing much build or taking too much time to retrieve those data so all of these things data engineer has to look into and they have to do the data modeling data modeling is in the sense like you are storing the data right so what is the effective structure the most performance structure the data can be stored obviously the ETL is like you can either be ETL or elt extract transform load or extract load transform based on their use case they have to do that and obviously like I said they're also building software Solutions focused for data problems so they should be knowing software engineering and data warehousing similarly like how do you vary those data in a huge level uh proper data governance in the sense who should have access to a certain type of data how do you separate one from another how do you store it accordingly they have to look into Android data Technologies here in the sense they also work with spark and all the different ones and distributed computing in the sense like for huge amount like we had targeted right we as we said we have like 94 said a bit of data available and let's say a huge organization has like it's not like they can store it in a single place it has to be distributed across many missions so that it's stored properly no data is lost and you retrieve the data faster and efficiently so for these things they should also be working with distributed computing distributed processing uh mild MPP multi uh massive parallel processing there are different concepts and they will be uh working or they have to know all this but also like I said it's not all of this a single data engineer should know all of this but most of the data Engineers know all of this based on their workloads and their responsibility and how their organization is working and the amount of data their organization is dealing with and for this the language is obviously it's like right and Scala Java R hadoops spark however those big data storage Technologies or processing Technologies used obviously they should also know you know cloud computing or like Cloud Technologies like how are they going to deploy these applications what are the different uh data values in data lake or data storage uh Services provided by these Technologies and how they can efficiently use them and obviously like I said like rdbms like normal something like MySQL postgres SQL Server versus like kind of this cloud data warehousing distributed solution like snowflake AWS received in AWS bigquery in Google so like I said again not everyone should know everything but they should have knowledge of some of it and then they can transpose the knowledge when they work with another tool so that's how uh this should be seen um by the data Engineers so these are the basic skill set that comes with data Engineers their roles and that's what they all do and like I said like I only majorly covered the uh kind of the core or like the bigger umbrella but there are also other ones that revolves around the same inside the data industry like I said there are some persons called Data architect the sense they work on the very high level they have to look from the top level of the organization C like for each and every Department each and every team how they can set up this data strategy they look more they work more like a blueprint of it and these Engineers have to build it a database administrator we are talking like different databases all this data has to exist into the company so depending on like the Technologies they use certain places they don't need a database administrators certain place they need data administrators or like the term of AWS administrator can be different as well because like I said they maintain the admin they are the administrators of databases they make sure like who gets access who don't get access or I mean that's also different from different company who kind of has this power or like who does this or who does that the next one is devops or data Ops uh it's like uh development operations or database operate weekends in the sense like all this we are talking about deploying applications and everything right so if the company is huge and we are having like huge number of applications running uh in the sense like the developers or the engineers who are building those applications they might not have the time to kind of monitor I mean they usually do it's kind of integrated environment but like their major Focus will not be like uh looking into if an infrastructure is running efficiently if something's going down going up or something happens something crashed so that's where this devops Engineers or data Engineers they come in and they kind of look into like majorly focus on if their major Focus would be like is the infrastructure efficiently used or is the infrastructure running is it breaking how do I make sure it's not breaking and how we make sure I have like an uptime of 99.9999 percentage of all the time so that all my services are running to give an example how I put right this happened quite a lot of time in the past years like two times or three two like a very like handful of times twice authorize WhatsApp stopped working if you have experienced that and I'm sure 90 of us or 99 of us didn't be my the first thing might not even came to our mind say thinking that okay WhatsApp has some issue no we would have thought like our internet has an issue everyone would have went to flight mode switch it on switch it off and then see like okay is it working or like switched off your Wi-Fi router and then switch turned it on again and then see hey is it working then later only you might have realized okay my other things are not working then okay that means WhatsApp has some issue so they have like that level of stability and availability so you even think your internet is not working because there is no way in chance that WhatsApp is not going to work because it always works it is being majorly done by this devops data data apps I mean data apps like specifically devops engineer who focus on data that's what they call data apps but basically these are the people like devops or SRE they call site labeled engineering are I mean they are like uh to be honest like I cannot put all of them together but they work on the same line of duty that's how I kind of can say it I'm not like exactly saying they are the same but they are kind of related and they do kind of similar work that's how I can put it then like engineering managers are basically like you cannot have like all the developers doing everything and making decisions and communicating with the business a lot of things are happening so it's more like an engineering manager or like mostly managers with like technical background they kind of think on a product perspective think like how their product in the sense is working how they can improve it what are the feature features that is going to help the organization all these things these kind of Engineers they kind of do and they do some little bit of people management look all the whether the engineers there and the team has everything they need to work efficiently and contribute to the organization so that's what uh these kind of roles do so now uh let's look into like how to start your career in data science to honestly like most of the points that is going to come the next it's like uh is already been discussed so when I say it's like how to start your career in data science it's not like I'm going to say like hey you do X and you will have a job as y that's never the case not only in data science any field it's not going to happen I can give you a roadmap or plan like which where do because like I said there are a lot of Technologies and a lot of roles and everything so you cannot blindly start okay I will uh land learn like everything at the time you cannot like achieve or like it will take you so much time if you start learning everything at at the time simultaneously or parallely you cannot reach a certain level for you to at least enter into the industry because there are certain minimalistic stuff or like the basic stuff and advanced stuff so I will give you a plan of like going from the basics to the advanced I'd kind of put it in a common path in the sense you follow them and it kind of like covers almost all the rows that we talked about and then so that you can decide which one you want like I said you cannot say like this is what to be honest like I wouldn't ask you to say that okay if you someone asked me hey which is the best role out of all the four on all the role I discussed I would say that is nothing like this it depends from Individual to individual so what I'm giving you is a proper plan where you kind of like try to learn all of this study them like try to execute them understand them then come back to the previous set what I said right what are the Technologies or roles and responsibility for each of it each of the role that I have described before you kind of then try to map okay this is something I really like this is something that is not my cup of tea so then you map it so that you can find like which is going to fit you fit for you then you give us freshers or like who are initially into your career you can try to check and see like okay this is something that I like so this is what I need to pursue or you can also decide okay this is not what I thought it was so the initially in your career itself you can switch to another role and see like okay so is this I'm really up to my liking okay so this is my like this is what I like this is where my passion lies so that you can take from there so uh that's what I'll be giving here so it looks very simple but honestly like it covers almost everything so uh this is like what I would say okay so it's not like what like it's not like what it's in pattern I was put something else okay it's a typo so it's like this is the plan I want to give you this is the road map and so I don't know what I why I typed that it's like this is the roadmap I want to type but it came there okay so uh the roadmap is uh initially like I said SQL is your entry for data without SQL you cannot do anything so start with basic sub SQL understand like how you write a basic SQL query retrieve data from some table some simple databases try to come if you go for any of these basic tutorials understand it try the query take some data out of it come up with some try to get a very specific you look into the data understand the data then you you yourself write your write some use cases for you then try to get the query and get the write the query and get the result out of it the next I said right SQL is it's for every role you need to know SQL the next is like Advanced DB Concept in the sense like roll to roll it differs in a sense like data analysts and data scientists doesn't need to look into those DB Concepts Advanced on like you said in the sense like how to store data efficiently how does index work how does a particular database Works how does it differ from a different kind of databases how is nosql and SQL Works internally and uh how is a row versus columnar database and how they are working efficiently or how is a data big like data technology various how it is differs from normal databases those kind of very Advanced Technologies is mostly will be helpful for machine learning or data engineering but you won't be needing if you are looking into a data analysis or data scientist then mathematics and statistics obviously it's mandatory for data scientists and it's kind of depending on the role and the organization data analysts might need or might not need so it's kind of like a s for data scientists and maybe for a data analyst but also like as for a machine learning engineer for data engineer it's kind of a maybe so it's not like definitely needed but it's good to have and programming language it's near for almost everyone like I said any program language required like python R Scala Java so these are the major programming language used across and definitely you should be knowing any one of these languages uh maybe like certain level for data analysts and data scientists some basic to uh some not Basics and intermediate level is okay but for machine learning Engineers or like data Engineers they should be like Advanced they should not be in an intermediate level they should be an advanced level of this and data visualization tools data analyst and data scientists may need it but machine learning engineer or data uh Engineers don't need it so visualization tools similar to what I did power Tableau power bi and all those things and next is Big Data Technologies so everyone might be need to make data Technologies and kind of group them together big data and ML and uh machine learning deep learning Technologies let's say first for looking into Big Data the level everyone might need to have a certain level of understanding in a sense like data analyst and data scientists should know to a basic level how they can retrieve data out of it how they can write some efficient queries out of it that's the level they want but for machine learning engineers and data engineer they need to even know one level further how they can how the data can be stored very efficiently and how that they can enable others to easily retrieve this directed data so those kind of things they have to know and machine learning and DL is very focused for machine learning engineers and for data scientists and the cloud Technologies like I said like it's good to have like understand different cloud data Technologies understand what are the commonly available Services across different ones so they can look into it for data scientists and data analysts it's okay for them to uh have them but for uh ml engineers and data Engineers it's a must they should know because like only very huge organization work or they have the capacity to have run their own data centers but for normal sized or like even because some of the bigger animation they decided okay it doesn't make sense for us to have entity Department to run our data center we will just go for a public cloud like AWS gcp Azure and then they just pay them and use the services there so it's obviously better for everyone to know like what are the commonly used features particularly focused on data and stuff and like uh data scientists should know those data centers should know this but data engineers and ml Engineers should definitely know how to use them then next is like a build your portfolio in the sense like I would say like take some open source data sets or some open source Technologies try to build application on yours like for you take a there if you Google for commonly say like for a data engineering project for data engineering product machine learning project or something they'll be commonly used use cases or a lot of stuff you kind of like have to build some stuff host it in GitHub uh so that like if it's in GitHub other people if it's really good other people use it or like uh and when you go for an interview the interviewer will know hey actually you know this stuff so that's how you used it so that they can they will kind of like they already know that you have it it's kind of uh how to put it it's good to have uh or I would say in my perspective it's a must have like have a proper GitHub profile well all this so that the users can kind of use it and like the interviewers can see it it kind of give you a competitive advantage that's how I see it cool so uh finally I would say we are almost nearing to the uh end of this so if you have any questions I also like we'll have this q a session after this for the next five minutes and if you have even more questions you can see my website my email you can uh ping me there email me I can try to answer it as much as possible and mostly I'll try to answer everything as Navan then here in the uh QA session itself yeah and like I said here is a reference I said I use child gbd to prove uh proofread my presentation and everything I put and yeah these are like credits to like the different ones I end up used so basically we can still here I put it here so let's uh go for Q a sessions like if you have any questions please uh put your questions in live chat uh so that I can answer them so I see like here Abhishek has asked is it like uh can I be a data scientist while studying CSE which is better btec data science or CSE so okay so if you are going to be like a data scientist I guess it if you have if you are already studying a very uh particular job in data science like course in b-tech data science obviously you can go for data scientists but just because you studied CSC doesn't mean you cannot be a data scientist right you study software engineering like the basics of uh computer science and everything then you come here and then you can learn the skill set like I said these are the things that you need to learn you learn them and then you can go but even if you take the uh beta data science or CSE both of the places you are deep diving into a lot of mathematics in CSE you have a lot of maths papers so obviously you'll be learning those and then you can Implement them or you can learn some courses in the side so what I meant is like just because you studied in CSE doesn't means that you cannot be a data scientist that's all there is nothing stopping you you just follow the roadmap I kind of showed you and then you can be a data scientist yeah can a developer be a data engineer will the experience be useful uh so uh like uh developer means what do you mean by a developer that's what I need to know so can you say like what do you mean by a developer is it like because even if you technically put data engineer is also a developer because he develops application so it's let's say if you are a let's say you are a front-end engineer a front-end developer or a front-end engineer and kind of uh mobile app developer and then you want to move into transitioning into Data engineering obviously you'll be it will help in a sense like definitely it will have a computer advantage in a sense you you will be doing a lot of coding so obviously in those kind of developer role you've already did coding in a different language in the sense you have to learn a different new programming language if you haven't worked with what is being used in data engineering and learn a different framework from what you have used there but it will definitely help you because you already know like the best it's not going to change the sense like let's if you are a front-end engineer like you write JavaScript if you write uh yeah web application okay software engineering like I said like uh data engineering is software engineering so let's say you write some back-end applications it's going to be still uh front and not fronted uh you will use the same knowledge and transpose that into your data Engineering in the sense you are going to use the same set of software engineering best practices it's not going to change at all right because like let's say you use a back-end or technology or like a friend and even play let's take very front-end you are using a full stack developer you use a lot of JavaScript and there whatever you are using right writing classes inheritance uh interfaces simple like for Loop while loop arrays they are going to be same in every single language right so obviously it will help you you can obviously transpose the only thing is like you need to learn like what are the technologies that are Frameworks that are reached for detentioning so data engineering depends like either you have to run like python mostly python or Scala depending on like which company you are applying for let's see like what are they usually generally using python ascala then learn those language and what are the frame of the use is it Apache airflow DBT spark you just learn those languages that's all yes I hope it answers your question so do anyone has any further questions please let me know so uh we are also on most two more minutes uh with any questions please type out your questions or we can end the session anytime soon yeah I hope that helps you now yeah thank you yes so uh we are on times or any questions yeah so uh there is jiten asking which programming language is mostly used for data used by data analysts so uh data analysts May mostly it's Python and SQL that's our go-to and especially in Python there is this uh package called pandas so those are the main like widely used for data analysts uh you don't know like if basics of computer science so let's say if you go for data so uh however if they obviously you need to know some basics of computer science the sense like I don't I'm not saying that they should know like writing programming language lighting code but they should know at least some Basics like what is a DB and how someone can write a SQL query so obviously some basics of computer science is needed for everyone that's how you put it because you are going to use a computer and do something so you need to know some basics of computer science you have to build your basic computer science skills understanding like what is a website or like how is it communicating with the house of say build front-end back end how is the data stored where is data store the very basic things you need to know and then a person can focus on okay this is how the data comes into the database and then they can focus on being edit analyst okay I write some exploratory analysis query get those data in so but I would say it's not like you cannot expect saying that I don't know any computer science so generally I can jump into it I mean you can learn you can take some courses to study those being a data analyst or a B analyst but even in this like there will be some basics of computer science and everything uh kind of involved inside it but I would say computer science Basics like basics of computer science is necessary uh we cannot skip that part yeah so any more questions I can spare another minute or two as we can end it here yes so I don't see any uh further questions let's end this session thanks everyone for attending if anything please uh email and you can check with me thank you