Natural Language Processing (NLP) Full Course – Beginner to Advanced [2026] | Learn NLP with Python

Key Takeaways

Hello everyone and welcome to the NLP full course, [music] a complete journey into how machines understand language. In this [music] course, we explore how computers read text, interpret meaning, extract [music] insights, and even generate humanlike responses. You will learn how NLP powers everyday [music] tools like chat bots, voice assistant, search engines, and AI applications we [music] use daily. We start with the basics of text processing, move into core [music] NLP techniques, and then explore modern advancements like transformers, LLMs, and generative AI. And by the end, [music] you will clearly understand how language models work and how NLP [music] is shaping the future of communication and intelligence systems. So before we [music] begin, please like, share and subscribe to Idora's YouTube channel and hit the bell icon to stay updated [music] on the latest tech content from Idureka. Also check out Edureka's [music] postgraduate program in generative AI and machine learning in collaboration with Illinois Tech. [music] It offers a unique opportunity to explore the cuttingedge world of generative AI [music] and develop advanced AI powered solutions. This program [music] covers in demand topics including machine learning, deep learning, natural language processing, prompt engineering, [music] generative AI, LLMs, rag, agentic AI and much more. Learn from industry experts [music] through a curriculum built around real world hands-on use cases designed to equip you with the practical and job ready skills. So check out [music] the course link given in the description box below. Now let us get started by [music] understanding what NLP is. Well, human beings are the most advanced species on earth. There's no doubt in that. And our success as human beings is because of our ability to communicate and share information. Now, that's where the concept of developing a language comes in. And when we talk about the human language, it is one of the most diverse and complex part of us. Considering a total of 6,500 languages that exist. So coming to the 21st century according to the industry estimates only 21% of the available data is present in the structured form. Data is being generated as you speak tweet and send messages on WhatsApp or the various other groups of Facebook and majority of this data exist in the textual form which is highly unstructured in nature. Now in order to produce significant and actionable insights from this data, it is important to get acquainted with the techniques of text analysis and natural language processing. So let's understand what is text mining and natural language processing. So text mining or text analytics is the process of deriving meaningful information from natural language text. It usually involves the process of structuring the input text, deriving patterns within the structured data and finally evaluating and interpreting the output. Now on the other hand, natural language processing refers to the artificial intelligence method of communicating with an intelligence system using the natural language. As text mining refers to the process of deriving highquality information from the text, the overall goal is here to essentially turn the text into data analysis via the application of natural language processing. That is why text mining and NLP go hand in hand. So let's understand some of the applications of text mining or natural language processing. So one of the first and the most important applications of natural language processing is sentimental analysis. Be it Twitter sentimental analysis or the Facebook sentimental analysis. It's being used heavily. Now next we have the implementation of chatbot. Now you might have used the customer chat services provide by various companies and the process behind all of that is because of the NLP. Now we have speech recognition and here we are also talking about the voice assistants like Siri, Google Assistant and Cortana. And the process behind all of this is because of the natural language processing. Now machine translation is also another use case of natural language processing and the most common example for it is the Google translate which uses NLP to translate data from one language to another and that to in the real time. Now other applications of NLP include spellchecking, keyword search and also extracting information from any doc or any website and finally one of the coolest application of natural language processing is advertise on matching basically recommendation of ads based on your history. Now NLP is divided into two major components that is the natural language understanding and the natural language generation. The understanding generally refers to mapping the given input into natural language into useful representation and analyzing those aspects of the language. Whereas generation is the process of producing meaningful phrases and sentences in the form of natural language from some internal representation. Now the natural language understanding is usually harder than the natural language generation because it takes a lot of time and a lot of things to usually understand a particular language especially if you are not a human being. Now there are various steps involved in the natural language processing which are tokenization stemming leatization the poss tags name entity recognition and chunking. Now starting with tokenization. Tokenization is the process of breaking strings into tokens which in turn are small structures or units that can be used for tokenization. So if we have a look at the example here, taking this sentence into consideration, it can be divided into seven tokens. Now this is very useful in the natural language processing part. Now coming to the second process in natural language processing is stemming. Now stemming usually refers to normalizing the words into its base or the root form. So if you have a look at the words here we have affectation, a effects, affections, affected, affection and affecting. Now all of these word originate from a single root word and as you might have guessed it is affect. Now stemming algorithm works by cutting off the end or the beginning of the word taking into account a list of common prefixes suffixes that can be found in an infected word. This indiscriminate cutting can be successful in some occasions but not always. So let's understand the concept of limitization. Now limitization on the other hand takes into consideration the morphological analysis of the word. To do so it is necessary to have a detailed dictionary which the algorithm can look through to link the form back to its original word or the root word which is also known as lema. Now what lemitization does is groups together different infected forms of the word called lema and is somehow similar to stemming as it maps several words into one common root. But the major difference between stemming and limitization is that the output of the limitization is a proper word. For example, a limitizer should map the word gone going and went into go. That will not be the output for stemming. Now once we have the tokens and once we have divided the tokens into its root form, next comes the POS tags. Now generally speaking the grammatical type of the word is referred to as POS tags or the parts of speech. Be it the verb, noun, adjective, adverb, article and many more. It indicates how a word functions in meaning as well as grammatically within the sentence. A word can have more than one part of speech based on the context in which it is used. For example, let's take the sentence, Google something on the internet. Here, Google is used as a verb although it's a proper noun. Now, these are some of the limitations or I should say the problems that occur while processing the natural language. Now, to overcome all of these challenges, we have the named entity recognition also known as NE. So, it is the process of detecting the named entities such as the person name, the company names, we have the quantities or the location. Now it has three steps which are the noun phrase identification, the phrase classification and entity disambiguation. So if you look at this particular example here, Google CEO Sundap Pichai introduced the new Pixel 3 at New York Central Mall. So as you can see here, Google is identified as a organization. Sundap Pichai as a person. We have New York as location and Central Mall is also defined as an organization. Now once we have divided the sentences into tokens, done the stemming, the limitization, added the tags, added the name entity recognition, it's time for us to group it back together and make sense out of it. So for that we have chunking. So chunking basically means picking up individual pieces of information and grouping them together into the bigger pieces. Now these bigger pieces are also known as chunks. In the context of NLP, chunking means grouping of words or tokens into chunks. So as you can see here we have pink as an adjective, panther as a noun and the as a determiner and all of these are together chunked into a noun phrase. Now this helps in getting insights and meaningful information from the given text. Now you might be wondering where does one execute or run all of these programs and all of these function on a given text file. So for that Python came up with NLTK. Now what is NLTK? NLTK is the natural language toolkit library which is heavily used for all the natural language processing and the text analysis. [music] So what is NLP? Natural language processing or in short NLP is a automatic way of presenting or processing human language. What I'm trying to say here is here we try to develop applications and services in order to understand human language. Some of the practical examples of NLP are Google voice search, sentiment analysis and many more. As I mentioned earlier, we use NLP to extract meaningful data from textual data, right? So, do you think NLP is a magical tool that when a text is passed, we get a desired output? Well, this isn't the case. But as a matter of fact, raw text input data has to go through various stages just so that we can perform operations on the textual data set. As you see here in the pipeline, a raw text data under goes data cleaning which involves steps like tokenization, stop word removal, limitization and many more. The next step is vectorization where we convert our text data into numerical format. Finally, based on the requirements, we perform the classification task. All right, now let's see few of these steps in detail. Starting off with cleaning our data. As mentioned earlier, here the goal is to convert raw text into clean text data. This involves steps like tokenization, stop word removal, stemming and many more. Speaking about tokenization, tokenization is essential for splitting a sentence or a paragraph or a entire text documents into smaller units such as individual words or phrase. Each of these smaller units are then called as tokens. Then we have stop word removal. Stop word removal in general refers to filtering words whose presence in a sentence make no difference to the analysis of our data. So why do we have to remove them? But we remove the stop word just so that you know our model doesn't get more complicated. In the next step we have something called as stemming. Stemming is a process of reducing a word into its root form. What I'm trying to say here is with stemming we're basically removing the prefix. For example consider a word giving. Right? So once a stemming is performed on giving the giving ends up becoming gift. Moving ahead we have vectorization. Text vectorization is a process of converting text into numerical representation. Here we end up creating something called as bag of word model which is a model that signifies or represents a text and describes the occurrence of text in that word document. Finally coming down to classification task. Text classification also known as text tagging or text categorization is a process of categorizing text into organized groups by using natural language processing. Text classification can automatically analyze text and then assign a set of predefined tags or categories based on the content. Now let's move ahead and understand an open-source tool called as NLTK. NLTK stands for natural language toolkit. This toolkit is one of the most powerful NLP libraries which contain packages to make machine understand human language and reply them in an appropriate desired response. So why do we need NLTK? You'll see NLTK has many built-in package to process our textual data at every stage. We can perform tasks like data cleaning, visualization, vectorization that will help us in classifying our text. So let me now move to my code editor and show you how we can pre-process or clean our data using NLTK. All right guys, as you can see here, I'm going to use Google Collab. Okay, although you can use any of the code editor like Jupyter notebook or Visual Studio Code, but I would prefer to go with this. Okay, so in the next stage now we need a data set, right? So where will I get that? So in order to get a data set what I'm going to do is from skarn dot data set import fetch 20 news right this fetch 20 news groups will give us a data set so let's quickly see how that would look like all right so if I have to execute this I just need to press shift enter okay so let's see our textual data now okay text data is nothing but we are going to create instance of our fetch 20 news group so we'll copy this and call it. So what would this return? This would return us a bunch right bunch of object. So let's now quickly execute this. If you're downloading this for the first time, it would take some time to download our data set. Okay. And this data set is present in this particular link. Fine. So this is done now. So let's quickly look at how this data would look like. Okay. So in order to do that we are going to use uh like type data text data and then you will see here this is a bunch. So let's now import our numpy and convert this back to a a list because we cannot perform any operations on bunch right we we have to convert this into a list. So import numpy as np. Uh let's execute this. And now what we're going to do is let's give it as raw text. So raw text over here would be equal to text data. Okay. Dot data. Okay. So let's print how this data would look like. Raw text. Okay. So as you can see here we have a huge amount of data set. Fine. And all of these are separated by commas. Okay. Let me quickly show you here. Okay. So as you can see first off we have a list and then we also have a huge sentences or you can say a paragraphs which are separated by commas. And make sure you don't confuse this to CSV. And now what we're going to do is as a list right we don't want to take the entire data set because it's going to be computationally expensive and apart from that it will take more time to execute right. Right. So, and to get a better understanding of what we're doing, we're going to take only first four paragraphs or sentences, I would say. Fine. So, in order to do that, all I'm going to do is I'm going to use slice operation have a colon and put four. So, let's see how this would look like now. Okay. So, as you can see here, we just have first four paragraphs of sentences. Let me zoom out here just so that you get a better view. So, yeah, fine. So, now we have our text data and we are supposed to start cleaning our data, right? So let's start doing that. What we're going to do is first off let's reduce all the upper cases into lower case. Okay. So let me just give here a text and give a heading step stage one convert into lower text. Fine. And now what we're going to do is in order to convert to a lower text. So we'll just give here lower text. So this is just going to be an array. Let's ignore this for now. So as we supposed to do this like we supposed to no matter what input you take right like whether it's for a training or a testing data set you obviously have to convert it rather than writing the entire code here let's give it a method so that you know next time if you want to lower our text we just call the method so def lower this is the name of our method and then we're going to pass data here that would be an argument okay and we have a for loop now for words in this data raw text fine for words in raw text what we're going to do is we have to append it to this part so let's give here as clean text in order to get a better understanding clean text stage one so I hope this is fine okay let me quickly copy this and we're going to append this part dot append okay so we're supposed to convert our words into lower case right so it's going to be str lower this is a built-in method and all we're going to do is we're going to pass This is words. Okay. So now we have a method. So let's call this method. So to lower and all we're going to do is we're going to pass our raw text input data. Raw text. Okay. So it's not st it's going to be str here. Let me quickly execute that once again. Okay. Perfect. So now what we're going to do is let's compare our raw text with this. So we have something like clean text stage one right cl. So let me execute this. So as you can see we have from which is capital upper case and we have what car is this and here everything has been converted to a lower case. Okay in the next stage we have tokenization. As I've mentioned earlier what we do with tokenization is whatever the sentence is there or a paragraph we convert that into either you know individual sentences or into individual words. So in order to convert this into words we have something called as word tokenizer. And to convert this into a sentence we have send tokenizer. So let me quickly go here and give this name of the next block as stage two. And this is going to be tokenize. Okay. And let's see how our tokens would look like. Fine. Same like before we have to write one array here. So let's give it as clean text two. And this is going to be empty. Right. And now what we're going to do is from NLTK dot tokenize import sentence tokenizer. Okay. And then we also need word tokenizer. Here I'm going to show sentence tokenizer only for the demo sake. We're going to use word tokenizer in future. Okay. So before this we also have to download something that's called as punkit. Okay. So import nltk and then we have nltk.d download we have something called as punkit fine let me shift enter all right so now what we're going to do is we're going to perform sentence tokenizer okay for words or sentence in obviously we need this data right clean text one so clean text one what we're going to do is sentence we're going to use send tokenizer as simple as that and we are going to pass this sentence Okay. And now in order to obviously we have to store this somewhere. We're going to store this in a new variable here. Sent tokenize. And this is just for showing you, right? This is just for the demo. That's why I'm not going to use clean text three over here or two. Okay. So we'll append this sent dot append. And we're going to append sentence here. Okay. So looks good. So let's see how this would look like. So as you can see here, right? Earlier we have a single dimension array. Okay. And here this each of these over here represents a paragraph. Now within a paragraph we all know we have multiple sentences. So as you can see now we have become two dimension array and each of these within this array represents a paragraph and each of the sentence in this paragraph has become a word or one particular character. Okay. So now we'll do word tokenize and we move ahead. We need word tokenization. So we'll add this to a clean text too. Okay. Because this is a part of this thing. So what we're going to do is we're going to perform word tokenization. So let me give a comment here. Okay. So now what I'm going to do is for word tokenize you know rather than writing this for loop I'm going to show you a simple and easy way. What we were doing so far is we are initializing this array here and then writing a for loop and appending it. Right? We can also do something called as you know list comprehension. Let me just show you what it is. So we have this right clean text two. Now in list comprehension what will happen is the for loop we write it within our list. Okay. So let's see word tokenize and we obviously want some kind of textual data over here which will fill it in a while. So over here I'm going to write for loop for I in clean text data. Okay. And now whatever words that come out of sentences I want them to be tokenized. Okay. So let us now see how this clean text two would look like. So as you can see here we have every word within this sentence converted into form of tokens. Let me scroll this up so that you can see how it looks. So still it's a two dimension array. You know whatever is there within this first array here within this array represents a paragraph. Okay. So everything over here has been converted into a single word tokens. Fine. So next stage right we want to remove some punctuations. You see in our data set we have some special characters, punctuations. We don't want these edus and dots. So in order to do that we are going to use something called as regular expression. Okay. So let me quickly show you how we can implement this using regular expression. So to do that what we're going to do is we are going to import regular expression. Fine. And now what we want is as this is a two dimension data. So we need two for loops. Unlike previous we had only one one dimension data but now it's going to be 2D data structure. Okay. So we'll create something here an empty list which is clean text 3 and then we're going to have a for loop. So for words in clean text two we're going to create one array here and now we're going to have another for loop to access inner words. Right? So for W in words now for regular expression what we are going to do is we're going to show a pattern. Okay. So S is equal to so E dots substitute. So wherever that particular character is there we want to substitute it with something else. So dots substitute then we have R which is going to be obviously a comment over here. Then we have a carrot symbol followed by W followed by S. We want to replace this with an empty string here. and then word. All right. And now what we're going to do is if is not null, if is not empty, then what we want to do is we want to append that word. Okay. Clean dot append w. Finally, we have to add this clean to our main array. Okay. So, let's define that array as well. Clean text. Oh, yeah. You have to give it an array here. Correct. Fine. So let us now append this to our array. So it's going to be clean text 3 dot append and then we're going to pass just this clean array here. Fine. Okay. So let's now see how our this thing looks like. So we have clean text three, right? So clean text three. You know according to our analysis all of these semicolon everything should disappear by now. Okay. So as you can see here, we don't have any special characters within our data set. Okay, still this is a 2D array but we don't only difference over here is that we don't have any special characters. We have only alpha numeric values. Right? Okay, this is great. So in our next stage we're going to remove stop words. I hope you remember what is stop words. As I've mentioned earlier, stop words is nothing but you know those words which are most commonly repetitive. Okay. So let me now in order to do that let me just show you import NLTK. Okay. We obviously going to download the stop words. Okay. So we have NLTK do.d download and we'll download stop words. Although we can find the stop words on Google. What we can do is we can copy those top words put them in the form of a list and if that stop word is present within our data set here make sure you don't include that in our clean text for stage. Moving ahead let me just give a title here for stop word removal. Okay. So now that we have downloaded our stop word what we're going to do is from nltk corpus import stopwards. Looks great, right? Okay. So now what we're going to do is we'll have clean text 4. Okay. Similar to the previous for loop, we going to have four words in clean text three. And then we'll just create an empty list which is going to be appended to clean text form. We'll have four word in words. If the clean text three contains this words, right, the stop words, we're going to eliminate that. Okay. So if not word in stop word dot words here we're going to pass the language. Okay if the word is not present in this list what we're going to do is we're going to append because that means that word is not a stop word right we're going to append that word okay and now we have to append this to our clean text for so clean text for dot append this list W. So let me quickly execute this. It would take some time to execute because it has to go through a lot of data set, right? So please be patient and let's see how it would look like. Okay. So this has finally executed and let's now see how this would look like. So clean text 4. Okay. So as you can see here we have removed couple of unnecessary words and we have just the important ones. All right. Now, so as you can see, we have removed couple of words over here. And yeah, so now moving ahead to our next stage that is stemming. I hope you remember what is stemming. Stemming is nothing but you know whatever word we have we have to convert that into a root form. So over here data processing is there after stemming it becomes data prep-process. Okay, that's what is stemming. So let me quickly show you how we can perform stemming here. Okay. So as I mentioned earlier we use stemming just to remove this prefix right. So for in order to perform stemming we have various types of stemmers. So we have something like porter stemer, snowball stemmer, lancaster stemer. So today we're going to use porter stemer right. So in this particular example, so let's get that now from [snorts] NLTK stem. Porter import porter stema. All right. So once we have imported this, we're going to create an instance of our port stema. So port is equal to portter stema. Okay. So just to give an example of what we're going to do here. Let's take three words. Okay. Let's take a list. Let's say list is A, right? So a has couple of words. Rather than having words, let's pass a list within our list. Okay. So we'll have port dot stem. This is how we call a stemmer. Okay. And here we're going to pass the word that we want to stem, right? And now we're going to have for loop for i in. Okay. So now let's pass couple of words like let it be like reading washing. Let's give one word which doesn't have a prefix like wash. Then let's give driving. Okay. So let's now print this. So this going to be I here. And what output that we I'm expecting over here is that reading should be converted into read. Washing will be converted to wash. And then wash should remain the same because there's no prefix. And driving would be converted to driving drive. So let me just print here and execute this part. I hope you can see this. It read becomes read. Wash remains wash and rest everything remains the same. Okay. I have done a small typo here. So it's going to be driving. Okay. So let me execute this once again. So as you can see here we have successfully removed all the prefix and they do make some sense. Okay, this is the case in case of port stema. But this doesn't hold good for when we trying to use lancaster stemer or snowball stemer. So let us now quickly move ahead and see how we can implement the stemer in our data set here. So we all know we need our loop here. Okay. So before that we're going to have array. So clean text five and it's going to be an empty list. So now we'll have a for loop another empty list. Okay. So now we're going to pass this list and we're going to append. So it's going to be w.append. It's going to be word. Okay. And now we're going to append this smaller list to this one. So clean text 5 dot append w. I hope this is done right. Let me quickly execute this. Okay. And let me see how this plain text file looks like. Okay. So as you can see here we don't have any more uh yeah obviously there's there are some errors. That's because you know this stemer might not recognize it. That's why we have multiple other stem. But most of the places you can see I know our words have been converted down to the stem words. All right. So I hope now you understood how to perform stemming. But you know as I've mentioned earlier we have multiple stemmers. We have like porter stemer we have like lancaster stemmer and each of those stemmers are unique in their own way. Sometimes what happens is when we perform stemming we get words which makes no sense right and that thing can sometime be really annoying. So in order to overcome that we have something called as limitization. So let me quickly show you what limitization is. So let me just give limitization here. Okay. So in order to get this limitization we use something called as word net. Okay. So from NLTK dot stem and this is a form of stemming but it makes sure the word which is being outputed has some sense okay import word net limitizer okay and we're going to create an instance of this so it's going to be word net or let's give word net w ne it's going to be word net limitizer okay and now we obviously have to download couple of packages so import nk And then we have nltk.d download word net. Fine. Perfect. So now in the same way what we're going to do is we'll create just limitize words. All right. So we have limitized words here. L e mm still be lemm and then we have an empty array. And then we'll have a for loop for words in clean text 4. It's not going to be five obviously. is going to be clean text 4 because this is the form of stemming, right? And now what we're going to do is just the same drill. We have W which should be an empty list and another for loop and we are going to append whatever is there. W.append the limitize words. Okay. So word dot limitize and whatever word we want to limitize. So it's going to be word here. And now once this is done, we're going to append this W to our bigger lim. So lem.append W. And let me execute this now that this is done. So let's see how this would look like. Okay. So LEM let's print this. Okay. And let me just execute this part here. Okay. We cannot print this. That's because it's saying data is too long. So rather than printing I'll just do this part here lem so that we can just see some glimpse of how our data looks like. Okay. So yeah so as you can see here although we are performing limitization but now the words make sense just to give you a brief insight right let's compare how our data looked earlier and how it looks now. Okay. So what we'll do is we'll take our raw text that is this part here and we'll compare this with our final text which is nothing but clean text four right that is after stemming sorry it's going to be clean text five okay so let's now see how it would look like so let me quickly print them print raw text okay let me execute this first all right and now let me print clean text live. Okay, as the data is pretty huge, what I'm going to do is I'm just slice this up over here. Okay, let me just take our first one data. Okay, this is going to be the first sentence. And as you can see here, we have all the words which have been tokenized. And it's unlike this part here we have everything which is it looks organized. Okay. So obviously we want to pre-process this data right because this makes more sense and it is more easy on a system to analyze and the classification would be pretty accurate as compared to what it would be over here. All right now moving ahead. I hope now you understand why we need to pre-process our data. Okay. So now that we know how to pre-process our data using NLTK let's see how classification of text is done. So okay in order to classify our text we use something called as navebias algorithm. So what is this nave bias algorithm? You see before we understand this navebas algorithm right let us see what is classification in simple words classification means grouping of data based on common characteristics. As you see here we have couple of figures right we have triangles circles and a square. And now when we pass this through a classification algorithm all of those get categorized into different different classes and it's totally based on the shape size and whatever other features are. This is a similar way how the navebas algorithm works. Okay. So the principle that drives nave bias algorithm is something called as base theorem. And we [clears throat] use base theorem to calculate the conditional probability. So let us now see the maths behind our conditional probability. All right. Then as I mentioned earlier, right, we use navebased algorithm to perform classification on our textual data. And navebased algorithm has something called as base theorem. Okay. And the way this base theorem works is that we have to find a conditional probability. So what is this conditional probability? You see conditional probability we can say it mathematically like probability of occurrence of event A when event B has already occurred is equal to probability of occurrence of event B when event A has already occurred times probability of occurrence of event A and this would be normalized by probability of event B. Okay, I'm sure you might be having confusion like what is this right? You see this line over here? This represents the conditional probability. Okay. So this is the conditional probability. Now what we are going to do is let's understand what is this right? So probability of event A and B. So let us now take event A to be like shopping and event B would be something like rain. Okay. So what is the probability of you going to shopping when it has already started raining. So this is what this means. Okay. Probability of A B right. So this represents and and you can also say it as conditional probability. So probability of occurrence of event A when B has already occurred. Okay. So when it's already raining what is the probability that you'll be going down for shopping and there are a couple of terminologies that you need to know when we are dealing with this. Let's quickly see that. Okay. So when we dealing with conditional probabilities we have couple of terminologies as I've mentioned. So this part over here is referred to as posterior probability. Okay. This is the most important part to be found and this is called as likelihood. This part over here probability of occurrence of event A is called as prior probability. As you can see by its name, right? Prior refers to something that has already occurred. Okay. And this part over here is the most unused part that is nothing but likelihood. Okay. We call this as marginal likelihood. All right. So now speaking about probability, let's see how this concept came into existence. So we have probability only because we have something called as random variables. Okay, this random variables give rise to randomness. Okay, to give you a better understanding of what I'm trying to say, let's take an example. Okay, so we have two bags here. We have say bag one and let's say bag two. Okay. And now what is happening here is in bag one we have balls. Okay. We have red balls. Five of Okay. So do you think probability exists over here? Obviously not. Right. So no matter whichever ball you try to pick out we're going to get red balls. And randomness over here is zero. Okay. So let's take one more bag over here. So this bag has like five red balls and then four blue balls. So do you think probability exists over here? Absolutely. Over here you can see if I try to put down my hand and pick up any ball. So probability of getting blue is nothing but the total number of elements right. So we have like five balls and this so it would be 9 divided by total number of blue balls that's nothing but four. Okay. So this is what is probability. So over here we have more randomness. Okay. This is how probability came into existence. And speaking about our conditional probability, let's try to derive this conditional probability equation. Okay. So the way we get this conditional probability is by having P of A intersection B. Okay. We all know this is equal to probability of A by B when B has already occurred. Okay. So this would be our equation one. Okay. Similarly, we know that it holds good for probability of B intersection A. Right. The reason for this is because P of A intersection B and P of B intersection A is commutative. Okay. So this should be similar. Only difference that we're going to have is the change in the values. So instead of A it's going to be B here times P of A. Okay. And now when we equate these two right we are going to get something like probability of occurrence of event A when B has already occurred times probability of B. And this would be equal to this equation over here. All right. So let's bring this down and then we'll have probability of occurrence of event A time probability of occurrence of A whole divided by probability of occurrence of B. So this is called as base theorem and if you see right so this is something which is very much similar to what we had over here. Okay. So this is what is base theorem and this is how we derive it. So now you might be wondering how can I use this base theorem for classification problems. Right. So just a quick recap as I mentioned earlier classification is nothing but you know categorizing a data based on its characteristics. Okay. So here what's going to happen here? We'll have something like you say we'll have a data set, right? So let's take X. So we'll have X data set. So this would be nothing but group of values. Okay, the text data. And then we'll have Y. Y is nothing but the classes and Y refers to the class. And what this class means is it can be like 0 1 and so on and so forth. So here let's take something like 0 and one. And here zero refers to being it not spam and this is spam. And here x would be nothing but group of emails. Okay. So now let's put this in a base theorem and see how it would look like. All right. So over here we have for base theorem we'll have something like probability that an email is spam. Okay. when we we already have the email is nothing but probability of this particular email being in spam class times probability of that email all divided by P of X. Okay. And similarly this is for spam email. And now for not spam it would be P of Y is equal to zero. Okay. Given X this should be nothing but given that we have a label of non-spam and then what is the probability of that email being here? This would be times probability of y is equal to 0 all divided by p of x. Okay. So this is how it would look like you know base theorem for finding whether the email is spam or not. So to better understand this right let's see what each of this represents. Okay. Okay. So let's take this part here. I'm pretty sure you must be confused what this part represents right. So what this part says is think that we have a data set. Okay. So let's this be our data set. Okay. So this is our data set and we have x values here and then we have y values. What this x values will have is nothing but emails. Okay. So this is nothing but group of emails and this would be a class. So if the email is spam it would be zero. If the email is not spam it would be one and then one and zero so on and so forth. Let's just take it as an example and this is something which is an unknown value. Right? So this part over here is an unknown. See both of these are same. Okay. So as of now let's just consider for a spam email. Okay. So we need to find a new email. Okay. So we'll be given like we'll have a test. So let's call this as a train data. Right? So X train. Okay. And this is the output. This is a class. So now what will happen is I'll be given a email. So I'll be like Janet find out whether this email that I'm giving you is a spam or not spam. So this is going to be like this. Okay. And then there'll be X test. But only difference is here we don't know which class they belong to. Okay. So what we're supposed to do here is we are supposed to train our model and figure out which class this email would belong to. Okay, so this is an email here. These are the question marks. Okay, we don't know what class does this email belong to. So what this P of X and Y is equal to 1 represents is when we are given a class Y. Okay, when we already know that a email is spam, that is this particular email. What is the probability of this email part of being this? Okay. And then we compute this part over here. And then finally what we're going to do is when we get this test data, right? X test data, we'll just say what is the probability of this particular email being a part of zero. Okay, zero means not spam and one means it's a spam. So we'll basically get a digital value over here. So for example, now let's take an example over here. So I got this X test value. X test is something which will be over here. So I got this extest value say something like free food. So now this would be represented in spam right? The reason is because spam is a keyword which is usually found in a fake email. So what will happen over here? So probability that a given email that is x test is spam. Okay. So probability that a given email over here so this email that is this email is not spam and is spam. So this would say something like this will have a high probability of being in a spam right? So this would give us a numerical value say something like 80 which refers to 80%. And this would give us like say 20 okay and this is nothing but 20%. So which among this is high. So obviously this particular value is high right. Therefore this email would be classified as a spam email. So this is how basically it works. Okay. So now in order to find this values over here you know that that is nothing but in order to find the value of our posterior probability we have to calculate likelihood prior probability and marginal likelihood. Although we can ignore marginal likelihood this is because we're trying to normalize it. So we can ignore this part. Finding the probability of this is pretty simple because all we need to do is find the total number of spam email by total number of emails. And similarly I would do it for total number of non-spam but total number of spam. But the only difficult part over here to find is likelihood. So let's now see how we can do that. So to start off let's say something like we are given some emails. Okay. So we have 100 emails. Out of this 100 emails we have 40 of them are spam. We know that these are the 100 emails and 40 of them are spam and 60 of them are not spam. And this not spam is represented by zero and spam is represented by one. That is nothing but y. Okay y is equal to 0 or 1. So how this would look like is let's say we have a table here and out of this table we'll have say x test which is nothing but the 100 emails okay so we'll have from 0 1 2 dot dot dot and this would end up till 100 and now at the same time we'll also have y is nothing but a class and these emails over here can belong to either 0 1 0 1 or anything but it should be either 0 or one and here it's going to be 0 or one it's just that we're taking an assumption so now what we're going to do is we'll be calculating our prior prior probability. So if this is our data set, our prior probability is going to be nothing but P of Y is equal to 1. So what this means is count all the spam emails. Okay, count all spam emails but total number of emails. Okay, so let's now see what would be the probability for this. So what is the total number of emails? It's 100, right? So it's going to be 100 over here. And what are the total number of spam emails? It's 40. So let's quickly write 40 over here. Similarly, we're going to do this for P of Y is equal to zero. Okay, here is going to be total number of emails. And here we're going to write all the number of non-spam emails. So what would this give us? This would be 100 which is the total number of emails and then we'll have 60. So this is how this particular part would look like. In order to give this in a mathematical form because you know we obviously we'll be putting this in the form of formula right. So in a mathematical form. So this is nothing but an average that is 1 by m summation of all the ones for y is equal to 1 or zero and here this i will range from 0 to n. So basically we're trying to add 111 over here. Okay. Okay, so this is how we calculate our prior probability and as I've mentioned earlier, we don't have to calculate our marginal likelihood and finally we are coming down to important stuff that is the likelihood. Okay, so this is the part this is the likelihood which is the most important part and the toughest part to calculate. Although it's pretty simple, you have to understand the maths behind it. So in order to calculate our likelihood, what we're going to do is we're going to calculate the probability, right? So let's see how we can do that. So we have this P of X when Y is equal to 1. Okay, what this means is when we have this email, right? So, we already know that email belongs to spam. We already know that email belongs to non-spam. So, what is the probability of that email belonging to this particular group? This is nothing but probability that email belongs to class one. So, probability that email belongs to class zero. Fine. So, now how X would look like? So just to give you a brief before we move ahead. X over here would be nothing but a email. So it will have multiple words and somewhere over here in the middle it will be like get unlimited 50% of and so on and many other words. These are called the features and based on these features we calculate whether this email belongs to a spam class or non-spam class. So how this would work? How this probability over here works? It'll take each of these feature let's take something like ultimate. Okay. So it's going to be like probability of ultimate belonging to spam. This would give me some value say 0.9%. Because it's high probability right that an ultimate word comes in a spam email. And then we'll also calculate at the same time probability that ultimate belongs to non-spam. So this is going to be less probability. You obviously are not going to use uh ultimate in your day-to-day activities, right? Or day-to-day conversation. So this is how it's going to be. So let's now quickly see how we can calculate for this. So now we have X right. So if X this is a capital X is nothing but list of words okay and this is nothing but an email okay and then small X represents the words which are there. So here it can be X1 X2 X3 X4 and this would end up to XN. So these are nothing but features or words. Okay. And X is the entire email. Then what we're going to do find over here is probability that X nothing but the capital X belonging to Y is equal to0 is equal to probability of all of these individual words over here. So probability of all of these words belonging to a spam. And similarly we can do this for spam. So when we have an entire email what is the probability that all of the words or the content of that email belonging to spam. So here we'll have x1 x2 x3 x4 so on y= 1. So this is how it works. Let's see the expanded version of this. Fine. Let me copy this entire equation here. Okay. And let's paste it on a new sheet. And let's see how we can calculate each of these. Right? So what's going to happen now? Probability P of X that Y is equal to 1 is equal to I'm just expanding this part over here. Okay, let me just erase this to give you a better insight. All I'm trying to do is I'm trying to expand this part. P of X1, X2, X3, right? So this is nothing but probability. You see this comma here represents and okay, it's an and operator. So probability of x1 belonging to y=0 multiplied by probability of x2 when y=0 that x1 is also not spam. So this is how it works. This represents and and then we'll perform multiplication and then we'll do something like this again. So for P3 probability that X3 that is nothing but the word this X3 belongs to non-spam category when we already know that X1 and X2 also belong to non-spam category. What I'm trying to say over here is each of these words are dependent upon each other only if X2 is considered as not spam only if X1 is not spam. You know all of these words are dependent on each other and the probability of them is holds true only if the other one holds true. So what's the issue with this is by the time it reaches this X and right it becomes pretty huge value and it becomes computationally very expensive. In order to overcome this we use something called as nave bias assumption and what this nave bias assumption says that when we calculate this probability right here we have calculated probability of x1 when y is equal to zero and then we have also calculated the probability for the second word according to navbas's assumption this word is totally independent of the first word. So the first word can have higher probability of being a spam and second word can have higher probability of being not spam but they are totally independent of each other. So this is what is nave by assumption is. So let's now see how this equation would look like after nave assumption. So what we're going to do is I'll write one below the other so that we get a better understanding. So if y is equal to 1. If an email we consider that to be a spam email only if p of the first letter or the first word of that email is spam is equal to 1. This will give us some probability and then I'll multiply that with probability of t