What Is Big Data | Big Data Explained |Big Data Hadoop Tutorial | Big Data Training | Edureka Rewind

Key Takeaways

what is Big Data so big data is a problem statement so anybody says I know that I have worked with big data would be a far fit story because big data is not a technology right big data is a problem statement so what is the problem that it defines what kind of problems Big Data defined so big data says big data is a collection of all the problems that arise is because of the large and complex nature of the data which can also put to ways in which you can process the data all in all big data is a problem statement that includes capturing of large amount of data curation of large amount of data storage of large amount of data searching large amount of data or searching from a large amount of data sharing large amount of data over a network transferring large amount of data analytics of large amount of data and visualization of large amount of data so any problem statement that is associated with large amount of data and categorized under these cases which is capture curation storage search share transfer analytics and visualization can be tagged as a big data problem so big data is a problem statement that talks about the challenges normal traditional systems fail or faces because of these scenarios how to capture large amount of data how to curate curate me massage the data if you're talking about you know loading a large B data a lot of data might not be correct you want you want to do a lot of massaging of the data you want to do a lot of cleansing of the data they all come under curation and then storage of the data nobody can give you a disk right that's what we said we are limited by Hardware right now so if you if we have large amount of data and you really want to store large amount of data for analytics later on that storage has to be provided by some technology if that is a problem that is a problem of how to store large amount of data then you are into a big data problem and then then you have searching sharing transferring analytics and visualization of the data all right so all these categorically are termed as big data problem all right so time for another story guys another stories if normally you know when I go to the webinar people tell me okay I understand what is Big Data you know you you have been talking about big data and know defining big data but from a Layman perspective give me a use case where big data is important for us all right tell me how we can use big data and Things become much more clearer when I tell them the story all right imagine that all of you are the managers of some Bank all of you are the managers of some bank and for this quarter your responsibility is to find out the best possible location where you want to set up an ATM now if imagine that you guys are the managers what would be your solution guys where do you set up an ATM what can be your your solution for that well if I am a manager I would say hey set up an ATM right in front of my house you know every time I going to go out every time I'm to come inside my house I'm going to you know go and use an ATM that decision might not be rational you know it will be for myself as a manager if I want to become a good manager if I want to make rational decisions I just cannot you know take a decision which is for my own benefit right I cannot make it so if I have to become a rational manager if I have to become a manager who thinks about the company what exactly do I need for making a decision so guys what exactly do I need for making a rational decision about putting up an ATM so you are seeing you know where less ATMs are present set up an ATM where there is a majority of the population Metro Station Airport depends on demand these are the ways of making a decision right making analytics in the requirement all right these are ways of making a decision perfect but if I have to make a decision so let's forget about let's forget about the bank right now in your life in your career in your personal life professional life if you have to make a decision guys what do you rely on for making a decision what exactly is that important thing that helps you make a decision which is rational only thing you need to know for making a decision is the actual data the related data if you know related data you can make decisions by looking into the data the data in personal in personal experience the data is represented as experience but these are data also right but everything is data if you have to make a decision it is nothing but data oriented so I as a manager I go to my bank tomorrow and I call up my analytics team and I tell them that your responsibility is next week Friday I want a report where I want to find out the best locations where I can set up an ATM in the city and that's what exactly you want to work on so I give that responsibility to my analytics team analytics team will go ahead and you know make a know discussion they'll set up a meeting and imagine that analytics team is you know based out of you know or just team of five 10 people they all sit together in a in a conference room and they start thinking all right we need to make a report we want to make a decision based report we need data perfect so they agree that they need data so now the question is we know that we need data where will the data come from so we have to identify sources of data all right so second phase is identify what the sources of data can be so what do you say guys you know what can be the sources of data I would say the first source of data would be my company's own database I have my customer customer database I'm going to pull up my database and find out the data the customer and their zip code all right so data internal to my company so this is perfect you know so I'm getting the data from my company's database all right so somebody else and somebody else in the team says like what uh vam says customer complaints so I'm sure you know a lot of people will be you know opening up their net banking opening up their customer care chat window and chatting with the customer care about and complaining about the lack of ATM right so yes I'm going to go ahead and find out logs of customers talking to customer care so these logs will be total free flowing text right this will be unstructured data total free flowing text Data you know normal one to one discussion over over a chat window these are total unstructured data but I'm going to fetch all the data from my customer care all right somebody might have also picked up a phone called a customer care and conveyed the lack of ATM in a locality so second one would be you know audio files get out audio files from the customer care logs for Source can be well you know I'm sure somebody might be going into social media and tweeting about lack of ATM right so I can go ahead and search for online tweets or online social media data to find out if somebody is actually you know facing problems unable to locate an ATM you know survey so of course I can go ahead and distribute pamphlets to everybody who comes to my my bank and ask them to fill out a survey where do they find a lack of ATM in the city or I can also have an online survey in my net banking every time somebody opens a net banking I can ask them to fill up a 30 second or a 10sec survey right we have multiple sources of data data will be coming from multiple sources if I want to make a rational decision I cannot rely on on my on making a decision only on one source of data that will not be rational decision if I want to make a rational decision if I want to really make a correct decision I have to look into the data that is coming from all those sources and collectively make a decision by looking into this data on an whole right a 360° view of the data I have to collect data from all the sources and then make a decision so my question is now this is where we have defined the problem question is somebody will say hey where can what can be a solution what can be a solution which can allow us to do that can can a database load data from all those sources it cannot it cannot store right it cannot store data coming from all those sources and have a unified view cannot have it you cannot have it so we all agree that this analytics team are limited by the progress of the software technology they are limited by traditional systems right traditional systems will not allow them to collect data from all the different sources and look into the data holistically it is impossible so maybe out of 10 somebody says hey I have heard about something called doop Hado is a new technology Hado is a solution that does give two things Hado is a solution that gives two things at the bare minimum one is it is a platform where you can load any kind of data and secondly it allows you to write processing algorithms on that large amount of data so Hado is a platform that allows us store all of all the data that you think about and it gives you a platform for processing the data that you have stored so these guys in the analytics team in that room they thought thought okay this is a correct use case for Big Data so we have identified that yes if you really need to have a platform a solution where you can collect data from all the sources and process it then there is no other technology apart from Hardo that can do it right now all right so what this team do is they collect data from different sources they store the data into Hadoop they process the data into Hadoop and then they create a report in Hado and the report will have maybe a chart saying the most important area or the top of the list area where you need to set up an ATM will be ABC location and the second in line would be xise location you take the report you come to the manager the manager looks into this report and says hey I have got what I need now what I do is I can go ahead and make a rational decision my decision would be set up an ATM in ABC location next quarter if I really want to set up another ATM I would go and say hey go and set up an ATM in XYZ location all right so this this is how big data had solves actual business problem so what does Big Data tries to do big data had tries to provide a decision- making capability by loading and storing large amount of data all right so it is a data processing platform that allows you to process large amount of data irrespective of how much of data you have in a very fast and timely manner because you even not worried about the processing capabilities right now and try to provide a solution which is used for business decision making so Hadoop is an analytical software guys it is not a point of salees software it is not a webbased software it is not a customer oriented software in that way it is an out andout analytic software it is used for making or deriving hidden patterns in the data so that the management the business can make decisions of the business it can help the business grow by looking into these store data all right so it is an analytics software used only to ensure sure that it can help make a decision on the business front right so it is a business-driven analytics software how data like video images used to store in Oracle before hadu well honestly guys honestly nobody used to save images and videos in Oracle what they used to do is normally they'll store the images and the videos in a cloud-based system and store the URL in one of the columns in you know in your database so database normally do not store images you know it's a very bad design normally do people do not do that they have a image server they have a some kind of a public based you know Cloud Solutions where they'll upload the videos and on everything they want and only the URL will be stor into into this rdbms you know I've seen people you know saving smaller images as blob Cloud objects but tremendously slow that's a very know strict no no into you know data modeling right so Nobody Does It vam says can you ask analyze audio visual files using Hadoop the answer is yes so what is Big Data we all have unidentified so you know Amazon handles 15 billion customer clicks per system you know or or per day so basically if you go into any other is it possible to view others question yeah unfortunately you cannot view others question and that is the reason you know every time somebody asks me a question I repeat the question I read it out loud all right but unfortunately we cannot yeah all right guys so what is Big Data we have been seeing Big Data in lot of different places and if you go to Amazon you go to Best Buy you go to any other you know retail websites all using Big Data stock market well New York Stock Exchange ncdc you know they're using you know stock they're using big data for analytics of stock trends and then 294 billion emails sent over every day so huge amount of emails sent and how to filter out the data so one of the use case is imagine that you have a very know large company and your company of course will be receiving a large amount of email you have to you know find out whether the email is a spam or not right so your mail server have to make a decision whether the email is a spam email or not a spam email so what do you do right you cannot open the email you know in a traditional system and just you know read the email and see whether it's spam or not so you have to do you have to provide a system which is a distributed system with large amount of processing capabilities so that on a real time you can find out whether the email is Spam or not all right so all these are use cases of Big Data we're going to talk about the other use cases of Big Data as we move along but all in all what I'm trying to say is Big Data is available in your surroundings you know when you look up in your in your surroundings everything you do everything you work with in today's world has a big data angle to it already right so even though if we think about Big Data as pretty new but it is been there for quite some time and people have been using big data for almost a lot of time right now and a lot of systems have been developed and you know have gone live into production with big data big data is very stable it is doing wonders all the companies all of the top 100 companies have already migrated to Big Data they are using big data and they're using it for true so one thing about big data is it is not a technology big data is a problem statement haduk is a platform nowhere we have mentioned that it is a software bundle nowhere we have mentioned that it is a tool so to be to be very honest I started my career in Big Data 5 years back and if I tell you about the things that I've learned in know last you know or or learned on the first year that I've joined Big Data you guys even do not know about the names because they have been obsolete by now so in Big Data environment guys you have to keep your mind open that every three months you're going to think about you're going to learn about and have to upskill yourself into new technologies so big data is not a technology it is a platform in that platform you going to have new technologies coming coming up the internal architecture the basic architecture is going to remain the same coming up as you know you cannot even imagine the rate in which Things become old and new things pip up right so in in Big Data world you cannot just you know hold on to your past law and say hey I've learned about it and I'm going to be pretty okay for next five years no in Big Data world the concepts are important when you learn the concepts the services the softwares they're going to change every 3 months that's how it is going on for last five years every 3 months there's something that new comes up and then data if you want to be in the technology world if you really really think about understanding the back end of the data system then big data is for you because Big Data you know is not reliable to a particular software there is not a single software that is associated with big data whereas it is just a platform and that platform gives you advantage of learning new things you know if you are Basics at Lear you can learn and you know get over with a new use case a new service very soon so what exactly is Big Data so let's look at this graph guys in this graph it talks about the trend of data all right so starting from 2005 people have been seeing a large amount of growth in data now if you classify the data you can classify the data into two different segments one is structured data what is structured data structur is the data which is already available in your system you know relational data system all your rdbms back system these are all storing your structure data if you look into the structure data you'll find that there is a linear growth of the structure data for last 10 years last 10 years there has been a linear growth of the structured data you look into the other phase of the DAT they have seen that data has been growing exponentially for last 10 years into this unstructured data unstructured data are all the data which are outside the scope of your stum are freely available like social media data email data you know all your you know internet based data every single data which is outside the scope of your organization are termed as unstructured data and people have seen that there is a exponential growth of the data what has happened in 2005 that you can see that there is an exponential growth in in this unstructured data that has happened in 2005 Facebook went live social media yes the same thing right so 2005 was where people went to web2 Z where people are talking about in all the social medias all the mobile applications everything was you know coming up so it was a huge boom in the data management systems right so starting from 2005 you see an exponential growth on the data imagine that you are supposed to make a decision in the business you know your business is supposed to make a decision can you rely only on the 10% of the data guys do you think you can rely only on the 10% of the data to make a decision will that decision be correct probably yes probably the decision will will be 10% correct if you're only looking into 10% of the data 90% of the time your decisions are going to be wrong similarly you cannot also rely on only the 90% of the data so to make a decision in today's world you have to rely on all the 100% of the data so you need to find out a platform that can process structur data as well as unstructured data that can combine structur data as well as unstructured data if there is a platform that can look up load all these kinds of data together then that decision making system that system will definitely help you make any kind of decision that you think about because decisions are backed out only by the data so data collection processing and analytics can be made you know sure only when you collect all different kinds of data and process together holistically right so this is the analy why you need to do big data one trend of the data you cannot just rely on the structured data or only on unstructured data you have to combine both these views together and see a holistic view of data uh VM says does company give more preference to analytics from structured data and then unstructured data yes and no yes and no all right so why yes and no again we know the percentage the data that is stored into your structured traditional system in your company these are only 10% of the data right these are only 10% of the data but most importantly these 10% of the data are all genuine data very important data whereas the other part of the story is 90% of unstructured social media data so although the number is 90% relative data from that 90% when you have to know crunch down the numbers when you have to analyze the 90% you will see that it is only 20 25% of this 90% that makes sense to you right so all in all a company has to give equal we to both but unstructured data will have to be you know analyzed more because you do not know what value that data stores right unstructured data social media data will not have large amount of value it will have limited amount compared to 90% the data value would be very less whereas the 10% of the data these are all actual really really important data so whole in all you know you cannot rely on only one phase and if you think 90% is good enough I can ignore my internal data well no you cannot do that similarly you cannot just ignore social media data I'll tell you a story guys again another story now imagine that you know imagine you know V you you go to a local shop that you have been going for last 10 years you know so VM goes to local shop that he has been visiting last 10 years and he wants to buy a bar of chocolate he really wants to buy a buy Know bar of chocolate he goes to the shop and he searches for a place where he goes and buys that you know picks up that bar of chocolate so every user who uses the system has an intrinsic and an extrinsic need what are the extensive needs of vam what can be the extensive need simple right the extensive need is the chocolate yeah the bar of chocolate is the extensive need he has gone in for buying that only what is the intrinsic need of vam intrinsic need would be because he's been going to that shop for last 10 years he thinks that the store owner will at least say hey Mr you know vam how are you doing today you know a oneone relationship or you can say hey vam you know I have this store counter open even though there are a couple of line people in the line he says Vino can you come here I can quickly Build You Up intrinsic need is he wants to get personalized touch from that store because he has been frequenting frequenting the store so much if I've been going to the same shop again and again I expect that the shop owner gives me a little bit more than anybody else that is the difference in today's world you cannot just keep your customers you know happy only by fulfilling their exter needs you have to C to their internic needs you have to understand what they want before they come and tell you that's what Apple has been doing right so Apple has has in going and be you know becoming such a huge industry right now company because they give something to you before you go and demand it you know that is the reason they have such a good and such a loyal customer following or customer base because you're thinking about the customers first right now right so your so in order to find out that intrinsic need in today's world when people are talking about online based you know shopping and all intrinsic needs can be found out only by looking into your social media and other data data which is outside the scope of your system outside your relational World your relational database will have extensive needs who has bought what but your intr needs can be only found out from outside the organization which is of course from social media and all other things right and that is why you have to combine your extensive need data which is social media data along with extensive need data which is a relational data and combine these two and find out analytics so that you can C to the customers better so no data is important no data is unimportant everybody has to C to customer need it it will come from both these sides of the story it has to come from relational world and it has to come from a non- relational the world also all right now you guys know Big Data you have heard about stories of big data I know you have been now you know thinking a lot about big data we imagine that you know all of us are here to become Tech arts in Hado right so we want to become technology architects in Hado imagine that we all are technology architects in Hado we go to a client place and the client says I have business problem so what do you say do you say hey I know Hado I do not want to need I do not need to know your business problem go with Hado I can do I going to you know tell customer that they need Hadoop no matter what problems they do have probably not right you have to find out whether Hado is the best suitable fit for their business problem or not you cannot just go and propose had to everybody it is not the way right you have learned Hado now you this is the latest technology you are learning if anybody says hey I have a problem you say hey Hado is a solution you just cannot go and propose Hado to everybody no matter what problems he has somebody says you know my email is not going to my manager he say you you cannot do that right all right guys you know so point is how do you know if a business use case is had to use case or not so IBM says there are four different ways of categorizing or tagging your problems as big data problem or not right if you're if if you can tag this problems that your client say into one of the four vertices then probably these problems are big data problems the first one is volume if the client is saying that my data volume is so much and my data is growing so proportionately that I think traditional systems cannot handle that amount of data if the client says that my data is growing humongously and in couple of months I cannot even imagine where I can keep the data then Hadoop is the solution for you so if the data if the problem is associated with volume then yes had is your solution second is velocity velocity says if the rate in which the data has been ingested into your system is so huge that your small system cannot handle that Vol that velocity of data then Hado is if you open if you write a web service how much events the web service can take per second guys not more than 100 right it cannot handle more than 100 requests coming and there is a sensor which is keeping on sending you data so nowadays people are talking about iots you know internet of things they're talking about sensors into everything you know they're putting sensors into everything and they have been you know push down every second if somebody is sending you millions of events per second do you think there is any traditional system that can handle that amount of data coming in no so if the velocity is a problem if the data that has been retrieved is so so huge that it cannot be processed or cannot be handled by traditional Technologies then Hardo is definitely for you variety you know if you want to make a decision and you know that the decision will be based out of lot of data and the source of data is varied and the source and the data itself is also varied like you have audio data video data log data text Data CSV data tsv data you have different kinds of data if your data is varied and you need to process them together you know that's the end of the story now you have to process them together then you need to have a system which can handle process and analyze this variety of data together so if variety is a problem then Big Data had is a solution and then verocity verocity says if the data that is coming from unsure about if the data is correct or not so you cannot save the data so verocity says if the data that you are pushing in the data might not be always correct because you're also talking about data coming from all those external sources you're not sure about the structure of data you're not sure if the data is correct all the time so veracity says crush the system if you are not sure about the kind of data you have you can still load the data into your hard BAS system so veracity says so and net net about velocity is in today's world guys normally if you talk about a transaction database any database you talk about every data have a shelf life right every data have a lifespan normally every you know every year older data will be archived into a tape right you going to know archive the data to the tape and lock the data away but in today's world you cannot just lock away your data because data is the most important thing for you data is like gold dust you cannot throw away the data so Hado velocity Point says you trust Hado with the data Hado says Hado is built out of commodity Hardwares we're going to talk about commodity Hardwares later on Hado is normally based out of commodity so the net net important part is setting up and maintaining had cluster is very very economical the money you spend on setting up and maintaining a had cluster is quite comparable to maintaining and managing your tapes so had says do not throw away your data do not lock away your data instead if you think that you want to Archive the data push the data into hard system and you run analytics on the system so that the data is not wasted if you have 10 years worth of credit card history of a person you can create models and see and create a you know fraud detection model maybe so important thing veracity says do not throw away the data at any point of time do not lock with the data use the data for analytics purposes you use the data and P the data into a system into a h based system so that later on you can do analytics if you are not allowed if you're not you know thinking about analytics at this point of time use Hardo only for storage but tomorrow you can go ahead and run analytics on that so veracity talks about the value of the DAT talks about the trust of the data and it talks about the ease of which you can load the data so even if your data is corrupted if your data is not correct still load the data because you're not supposed to throw away any data right so if your if your client talks about his business challenges and he falls under any of the four vertexes which is volume velocity variety and velocity then you can go ahead and propose him saying that hey you need to have you need to have Hadoop as your solution how velocity is different from validity well validity is a common term right here we talking about the four vertices of tagging a problem so here the velocity problem is you are supposed to you know archive your you know your data at let's say after every year you are doing this process right everybody archives the data your transactional database cannot handle years worth of data you have to Archive the data had says hey do not archive and throw away your data into tapes because if you load the data into tape you cannot use them again so do not throw away your data you move the data into Hadoop use Hadoop as an archiving platform so that anytime you want you can run analytics on that data what kind of data it can be it can be 10 years worth of your credit card data imagine how important that data would be guys 10 years worth of a customer buying and selling patterns or buying patterns from credit card so ver veracity says trust Hado with the data use Hado with data do not throw every your data