NLP aspects in Telecommunication Industry

Key Takeaways

hello and welcome everyone to another session in the data R series we are thrilled to be here with you this evening for a session full of action-packed learning I am Priya Pandey part of data science team at analytics Vidya and we have rishikesh kokare who will be moderating the data session with me for those who have joined us for the first time a brief introduction to the data sessions the data R is a series of webinars conducted by analytics Vidya and led by top industry experts it is a fun way to understand the concepts of data science from leading players in the data Tech domain and as the name suggests it is one art dedicated to data we are hopeful that these sessions are going to be great source of enrichment and value-adding for our community members now on to the our special today which is like now on to our session today which is NLP aspects in telecommunication industry national language processing NLP is an art and unique technique for analyzing modern data like speech text noise Etc it has changed the game of telecommunication industry remarkably in this data the speaker will explain how machine and deep learning along with NLP can make sense out of this modern data she will elaborate how modern data processing is done and how telecommunication Industries are using the NLP technology to draw meaningful insights for their benefit I hope you all are excited to attend this data with us before we kick things off and I hand it over to our speaker a quick recap of the housekeeping items we are recording the session and the recording will be available on our YouTube channel you can find the link in the chat section please use the Q a section for asking any question you might have during the session and we will do our best to answer them as the data progresses or towards the end lastly we will share a feedback poll towards the end you you all are requested to kindly fill that up before leaving the session now on to our speaker in this session of data R we have sakshi gujral with us sakshi gujral is currently working as a data scientist at concentric she is also pursuing her PhD from triple it Delhi she is a gate scholar and a ugc net qualifier as well and most importantly she was Elimina of drdo that is difference and this is a research development organization so over to you sakshi the virtual stage is all yours thank you and then thank you so much for the lovely introduction and a very brief for for the course of the data art uh a very big thanks to analytics Vidya who has invited me for this session uh so a very big hi uh to all the audience here I can see people from various countries have joined thanks a lot uh so let's let's begin the session um this session would be on NLP in the telecommunication industry so I am coming from the industry and it has been almost like 4.10 years to me working with the various kinds of the data and out of which one industry that I still feel will be growing um and have variety of modern data is the telecommunication one so with this I'm going to share my screen so that we can begin the decision any doubt in between any questions uh you people can post and I would try to answer I hope my screen is visible yes ma'am thank you so national language processing in telecommunications so so I would say past three years has witnessed a lot of changes in the data so earlier we used to have data generation from the conventional banking sources finances and uh um like very conventional sources now but with the increased use of social media uh the amount of the data has increased a lot now every domain I would say is having the data and telecommunication is one such domain which is growing at a very fast speed and this data I would say would be a new oil factory in the upcoming years and it it can provide us hundreds of insights um which can lead to millions and trillions of the insights from this data okay so national language processing is the data where uh the way we speak and the way we try to speak in different Regional language such becomes the national language so national language is no more restricted to speaking only English now uh if I talk about the additional language especially in the uh countries like India where uh where I would say only Hindi is not the language Hindi keeps on changing its it speaks in every different regions so that also becomes as part of a study so very frequently across all the groups national language processing is done on the English language so here in today's uh data R we are going to cover such Topics in the English language itself so I have structured in a way where I will be introducing you more of the nlps stuff we will talk about some applications of NLP we'll talk about the basic pipeline that that would be a little bit technical of NLB then I'll come to NLP in the Telecommunications overview and at the last 5th and sixth point that I have noted here I would be showing you some python code so that you can get an exposure that how we can do some interesting stuff with the help of the code as well so we'll cover some stuff with the basic machine learning and finally the advanced deep learning algorithm as well so I would try to I would try to cover from basic to Advanced like the moment I'm going to wrap it up you'll have a knowledge about the basic NLP stuff and you will try to learn more uh for the advanced one as well okay now uh with this diagram let's try to study how this language language processing is generated and how it can be and how it can be understood so it's it's like we can see here there is a person who is talking like where what is your return policy so that conversation is being recorded in the form of the text data now comprehended what is being said the way we are going to understand it in terms of the computers that becomes the natural language understanding now what after this like we have understood we as a human have understand but how to make our machines understandable to it we need to do some code we need to do some processing so once we have done this process saying once we have done the code what next like this is the data we once the data has been processed we can see what is being talked but now how I can make use of of this data to to gain some insights out of it then machine learning deep learning comes into the picture so there are there are two pipelines machine learning and the NLP so one NLP is used to capture the data for the natural language processing and once the data is being captured to make some interesting more interesting stuff so that we can go for the automation we require machine learning and deep learning so here in the diagram we can see machine learning can help us to determine the right response and further with the help of training the machines we can go for the natural language generation so a simple example would be um a bot a bot to whom I have trained certain English words now it has learned with the help of the machine learning algorithms not this what itself can start producing some national language the moment we are going to write something it it can it it is it is capable enough to generate the entire sentence that comes the natural language processing going towards the natural language generation so it does NLP turning to nlg now uh I have included a slide just to have the real life applications where you can link NLP in a day-to-day basis as well so very very simple example is the sentiment analysis so whenever you go to restaurants or go for a movie you always find some feedbacks where you are supposed to submit few stars or maybe you're supposed to write some reviews about a delivery boy or the food that becomes a sentiment analysis so it it actually helps Industries so it's just not the um summation of stars it's it's easy job for us but how if I see from the industrial point of view from the business point of view um at the back end all the biggies are trying to analyze the data their service their uh I so if I if I talk about figures matter uh the customer feedback for a delivery executive is really helping them a lot to um to track the performance of your stuff so this is how a simple sentiment analysis is saving the money of many of the biggies and they are ready to spend a lot on doing the sentiment analysis trying to improve it with the passage of time now uh beneath it we can see a person is talking to Alexa so I think Alexa Google every day we try to speak we try to set the alarms we try we try to change we try to use them so that we can change the music in two tasks for us so this is another beautiful example of NLP which is used in day to day life and natural language processing Market on the right hand side corner the top one uh an Easy Market Trend as well as so wherever there are some comments uh so there is textual data involved offline or online that can be processed with the help of the national language now how important is NLP in Mobile so any any answers for this I would like to see some answers in the chat window for for this question anything that you people can think of I'll stop for one minute jackpot analyze people conversations Google translate nice I'm getting good responses social media text assistance machine translation WhatsApp very nice for purchasing items calling numbers OK Google customer satisfaction wonderful vertex AI the advanced content moderators ticket classification CD wonderful so I have got lots of answers for this thank you so much yes uh so you people have written already that how important is NLP in Mobile so um the recent times where mobile phones have improved a lot in terms of both hardware and software we can we can say this mobile phones are now the wallet of data whatever activity we are trying to do Google is analyzing and other other features or apps is also are also uh analyzing this data why are they analyzing um so that they can make use of this data to improve their sub and out into the market the new automations which can really help us out now let's try to be a little bit technical about the NLP stuff so we see a lots of data on the social media so not only it's like social media data or data which is present into the digital forms can be an NLP applications if we have bunch of documents itself uh in a in a form of a PDF where we like it's on a paper and we try to scan them click a picture but we can still use the data from there and get it analyzed with the help of NLP itself so number one example is finding appropriate documents in certain topics from a database of text now if we see um searching queries if we try to search for a particular document using the conventional algorithms uh it can really take lots of time searching is a cost cost to to hardware and to that I mean there is there is always time in space complexity involved when we are searching but if we know the correct document like some keywords or with the help of NLP if you try to do the same searching operation uh the operation time would be less secondly is extracting information from messages or articles on certain topics for example building a database of all stock transactions so NLP give us the very sophisticated applications to write the regular Expressions so regular Expressions can be written without NLP also like some people who are using apis gateways uh so those people make use of the regular Expressions to get the authorization tokens but if we talk about the NLP stuff and the associated libraries for writing down the regular Expressions those are like quite powerful especially if I talk about Python and r third if if I have to translate the documents from one language to another for example the producing automobile repair manuals in many different languages so in if we make use of the NLP or that it would be again efficiently used so the these are some of the examples where some of the work has been done and uh researchers researchers all around are trying to improve the models model performance so that they can bring out some more new techniques and improve the accuracy of the existing work I think someone has raised the hand is there any doubt now uh it says summarizing ticks for certain purposes so instead of going for a thousand page documents I would try to extract the relevant information from this document and then uh I would prepare this summary so that would not be a manual processes that would be again an NLP process that NLP uh gives me and a power so that I can extract the directly keywords from all those Corpus thousand page reports and get the important uh words out of it question answering systems so modern days question answering systems where NLP is used to query a database those have also been upgraded from the NLP point of perspectives now lastly I would like to introduce that in the telecommunication domain uh automated customer service over the telephone so uh we all tried to call this uh customer service not only in the for the telecommunication one but for the other uh other services like sometimes our cards are not working or or some Airline so they start recording the text and that text is being analyzed so that is again a very useful application and it is still a very important research area which uh Engineers scientists are trying to improve the accuracy is trying to make it more efficient foreign so this is a very general diagram which would give you a gist for all those who don't have worked I mean exposure to NLP before that how NLP systems looks like like what are the stages into it so we get a text usually the natural language process text contains lots of noise so when I talk about noise it means messy data the data can be data can contain the repetitive words it can contains the hyphens at the rate some emojis um and special characters so that needs to be pre-processed because anything any wherever I am going to apply machine learning I have to make sure that the data which has to be given to the machine learning models though that has to be in the highly sophisticated represented firm otherwise uh model will not be giving us the good results we are not going to train it in a proper way so I would be doing some text processing sorry and then uh second part is the Eda Eda is exploratory data analysis so very first thing whenever we get any type of data whether it's on medical data or an image data or a video data we should try to look what is contained into that data the insights the actual Trends this is how uh data scientists work what is actually contains what is the distribution of a variable uh because we should try to deliver to the clients that uh this this is your data it contains so and so Trends now from this Trends I can try to bring out some more insights or we can do X even task onto it now when we talk about Eda Eda once you are giving the insights your next step should be the feature selection now if I say I have a CSV or an Excel that contains thousand columns not all thousand columns are useful to me so I would be selecting some subsets of call columns that is a Feature Feature selection for that I use some of the embedding techniques so here is there is an example of what to make what to make is an embedding techniques which tries to find out or extract the important information out of the text and then uh we come to the representation part so representation part in terms of the NLP or any other Advanced Data modern data is quite important reason is that if we are representing in our data that would help us to improve the modeling part so if if the data is messy too many dimensions and uh it's not in a proper form so model will not be able to process it in a way it should be so this representation itself is a good research area where lots of research has been going on especially for the text and the image data as well as sound also now next comes the modeling part so here uh depending upon the task like if I want to do classification if I want to do some regression if I want to do clustering I would be selecting the model and depending upon the size of the model itself I I need to take the decisions whether I am going to take the Deep learning model or a machine learning or other stuff and finally it is a deployment part uh but make sure that all machine learning models are not Deployable because these take cost cost again into the times of the time complexity and the computation complexity so sometimes it happens we are not actually deploying them we are just keeping it and running it iteratively like a quarter or six months to get the data dumps and process the model and then analyze the results and then make changes so that's also a way some of the organizations are doing so I pause here for for a moment and uh let me check if people have any doubts okay so I have received a question what is the meaning of Eda I mean in Chrome for what piece okay so probably I will take the doubts I think at the end of the session you people can post it yes okay moving on to the next slide okay so this diagram if we look for the very first time can be a little bit scary but this is important to understand because later on I am going to show you the from the coding point of view how does it works so number one step as we have read data pre-processing comes into the data cleaning and then it comes into the tokenization so uh let me give you a very simple example let's say I have a sentence RAM and Sham are playing so this is entire sentence now this entire sentence cannot be processed directly by a machine learning model what I have to do I have to first of all break my entire sentence into this into the words that breaking or breaking of entire sentence into the words become the tokenization then I try to bring out some important information from those tokens then I do POS tagging POS tagging as part of speech stacking that means just like in English I have got uh pronouns nouns verbs I do the same way but I would say that four and fifth these are the steps we don't usually use in all the use cases depending upon the requirement it's it's just an Eda part fifth is deemed entity recognization so once I have broken down my sentence into the words whatever I will be doing I'll try to find out the important keywords in from the uh from the sentence uh it can be not only a sentence I mean it it would be definitely a document where I would use the frequency of the words will be given me given by the named entity recognization the important words I can have it from there and then uh depending upon that I will try to do some more computation as we can see uh at step number six and seven that I will be doing some limitization limitization is like RAM and Sharma pain so playing itself is a token but I cannot use ing into it because that's that's giving an unnecessary information to my machine learning model so I'll try to convert my playing I uh playing board into the plane the root firm so that is being done by the limitization so we have got couple of libraries in Python where I can do uh various techniques where uh limitization can be formed and then I try to extract the um features that does like engram extraction so it means like from RAM and charm are playing from from this particular sentence how many words I want to analyze at a single like two words would be a diagram extraction if I want to take three words simultaneously that would be a trigram extraction uh similarly like this and then I would be performing some tasks like classification clustering depending upon the requirement so in any case if you people are not clear with this slide that's not a big issue because it's it's more of the technical stuff uh I'll try to show you for the coding part that would be more clear to you at the uh end of this CDs now let's talk about NLP in telecommunications so so uh telecommunication is receiving variety of data uh from their grades from their customer support office so uh this is one of the industry which I have chosen today to talk about because this is receiving the data in the form of the time series data some textual data also the images of the towers so numerical data great data some some data like this so it is receiving variety of data to be analyzed so um I have worked on one of the one of the clients which was uh not only one of the I mean I have a decent uh work experience for working with the telecommunication client so I would like to share the same information with you that how we were trying how we try to solve these use cases for them and that actually helped the business out of it so as we can see from the diagram uh we can see the very first use cases system of Records so um they have got some most of the telecommunication clients have their CRM systems where they can capture the data now this data can be coming from the chatbots SMS social media voice and the video chat and also in terms of the marketing data so uh campaigns can be run off to collect the marketing Trends in these strategies every every where the data is we can see for this particular diagram is more of the NLP based data now uh once it is being done uh they have got some iot stuff also like some wearable device data cloud data so every data is being collected uh at the central level and then to analyze this different type of data which is actually a textual data most of the time they try to build out the um Pipelines now uh one such example I would like to give so uh there there was a client a telecommunication client of mine who wanted to make sure that whenever the network parameters it is providing to to the customers is aligned with the post they are posting onto the social media so it was a case that uh if a telecommunication provider is providing me or Internet service suddenly my internet internet has stopped working now people start posting uh giving their hashtags that um let's say hashtag ABC that this customer this provider is not providing me a satisfactory Services now what happens if such type of comments are being recorded onto the social media this can actually hamper the business so there is a loss to the industry now uh what what to do in that particular scenario there are two things oh um grids that are supplying the internet and to track the social media or any other platforms apart from all this Force which I've mentioned here that whether they are aligned or not so this was the use case that was being approached to us and we provided the solution in such a way that uh we combined the data for from the um one of the social media portal uh that is Twitter and also the data provided by by the client into the form of CSS that was a network parameter data and thirdly we had another social media date another social platform that contains the again again the data similar to the Twitter point of view so I'll tell you about that also so first of all let's see what kind of data we can have from due to term so single tweet can give me a plenty of information like who has posted the geographic location the sentiment of the person along with the Emojis so emojis itself is a another research area where uh emojis names detection uh this can happen with the help of NLP because they give us these emojis can be mathematically converted back into the text and the further numerical values and then they can give us that how happy or sad a person is with the help of these services information for Point information about the operator being to eat and Subs tweeting that is metadata about it does retweets and live tweeting about these short events so these are the seven points I have included here uh just to give you what important data we can get it from Twitter uh there are a couple of other columns also that you can include in your code and get the relevant information so what I have done for this particular use case hi who's the customer provided the data that was being provided by the telecommunication client tweets I have extracted from the Twitter and there is one more platform called down detector.com that is like quite famous uh within us that maintains the records of the problems and then uh this website allows people to post the comments whenever there is a downtime so the data from the downdetector.com and data from the Twitter actually helped us to get an aligned and to give a good picture that whether there is a network failure or not so we chose a particular I chose a particular date on which there there was a network breakdown so it was somewhere around 19 July and then uh I tried to subscribe the to the data for that particular date I also went to downdetector.com and then check these charts for there so we can see here 83 percent of the people have reported that internet was not working 10 percent reported TV was not working and five percent said it was a total blackout so this is how uh this is how this base was built up now we did the Twitter sentiment analysis for this so what we did we do we tried to observe from the tweets when the outages happen and then it was the using the Twitter API and then uh I did a sentiment analysis onto it with the help of a machine learning model that is logistic rotation logistic regression is more of the uh statistical model I would say so I'll explain all the stuff in terms of the codes also once I'm done with this particular piece now with the help of this uh Twitter sentiment analysis how it benefited the client how it benefited the business so uh it gave us an approach that if the models are being trained it can give us the um time reactions that Which con which vectors are important for the network providing services and uh if we provide the good Network Services definitely it would improve the customer experience so it can also give us the idea that um so and so at the peak hours where the internet requirement is more so in that particular scenario organization can take a step that I need to for I need to supply more of my internet bandwidths during like evening Pickers or morning Pickers wherever the traffic is more and then it can definitely improve the business and uh what was the outcome of this I was able to find out what all top 10 frequent words that are being used by the customers and then I had a model which can classify the tweets like whether a customer is happy or not with the service with the 82 percent of accuracy and uh if we can see here out of the Thousand tweets so it first it was tested on a very small data we just came out with the conclusion that 8 40 percent 840 people weren't highly Satisfied by this so I take a pause here so I just wanted to know uh um from the response that uh are you people following it getting it or is it like making sense to you okay thanks a lot so uh before going forward I would like to show the small python code that how we can do this I'm going to share my screen back just give me a moment yes so now I will be showing you a python code that how I did the scrapping of tweets so this is the Google collab uh the online editor that I have used so there are a couple of libraries that I have used here and set up here just to save the time out of which one was a preprocessor so and then Twitter API so before scraping the data from the Twitter I need to make sure that I have a Twitter developer account so Twitter provides the open API uh to Twitter provides the developer site from there from where you can get some these are the credentials so you make an account over there your tokens will be generated this tokens would be would be having some like stand that is there they can expire as well so whenever you want to scrape the data you need to make sure that your tokens are alive once you are done with it uh so python provides 3p from where you can extract the data and once I am done with it I can do a search words search words give me um method that hashtag this word I want to fetch the tweets from this and then uh with this piece of code I can specify in which language I require and how many count I mean how many tweets is a single hit of this API I would be requiring and from which date so there are a hundred of other things you can do with this when you are hitting the API you can include more of the columns so I have just used it the way I want nothing else so Twitter developer documentation can really help you it can give you longitude latitude and other information you can always use it and then with the help of CSV file CSV writer uh python package I am able to extract the switch and save into a CSV so I would like to show to you the similar way I I try to shape this data from the Delhi violence so let me show you how the streets looks like so I can see here these are my tweets so date is coming username is coming from which location it is coming and these are my tools so this data is quite messy if you can see here there are lots of candles there are new lines there is a time so we cannot use this data directly we need to pre-process process it and then find out the distributions of the words uh it's it's not always the medical data that can be visualized we will be we have got lots of opportunity to work on this textual data to get good very good insights from this so this is the small example where I have showed you how to just grab the tweets now I will be taking you to the another notebook where you will be uh seeing that how I use uh the streets to classify the data that I have showed to you in my presentation for one of the Delhi communication client so I'm sharing my screen back yep so this is a small code we have done logistic linear regression is quite basic model since the data set we wanted to test it was a solution for the solution point of view we should not go and apply very complex models we should go for this basic statistical model because they give us a very good results these are not the black box models these are like white box you can see how statistically the results we are getting so uh once I have scraped this data I have splitted it into the train and test CS sweep and this is how my data looks like so there's a column called label which is one and zero so this this I would say the tedious task of the NLP um and to get a label data so how to solve this task number one approach is that once you are having the data and you want to label it label means whether a customer is happy or not one is Happy zero is not there can be other other stuffs as well where we can find out some more categories like a multi-class classification problem where we where we have tons of data and we need to classify the data into the five clusters I would say there also you require a label data number one is you can sit with the domain team the operations team that can really help you to guide from the domain point of view that which data Force should fall into which category number two is you can write some code so with the help of the uh Python nltk and species specifically for this passy I would say Spacey can give me um this entities creation entities creation functions which can help me to get this labeling thirdly we can uh make use of some paid resources which can label the data so very famous is Amazon Prime Services which can help us to label these data because label the data at first point is costly but if I look for a long term uh Roi it's good if if I'm if I have trained my model with a very good quality of data definitely uh and I keep on doing some task iteratively it it will reach to a point where I can get the good prediction out of it so this is how my train data looks this is how my test data looks so here we can see label is having any end right now because I want to test it I want to make my model to do the testing onto it now since this streets are highly High highly messy and highly dirty I need to clean it so we write some code so it's a function called remove pattern where I try to remove the handlers which are not making sense to me then Twitter handlers have been removed so I'm not going into the much detail of this uh this is how I am showing you that it can be done uh then with the help of these are called some regular Expressions these are also regular Expressions which can help me to do the cleaning of their this data and now we can see how my tidy to it looks like after cleaning this so this is again an intermediate process intermediate step I can still clean more of the data now this is token so we have talked about this in the NLP pipeline so we have tried to split our data into the small into the words that are called tokens here and then sorry we talked about the limitizers limitizers and streamers these are two approaches for making for extracting the roots out of the words so we did here the same way so that we can we can just like very simple example doing his hair it is do now because ing given to the model will not give me any good information so that's why uh with the help of stemmers or the limitizers I can do the same task only difference between limitizers and the stemmers is that one gives you the words that is a dictionary meaning and one can give you any abruptly words so once we are done with it we go for the vectorization this part vectorization or embedding this part is like when I am trying to convert my text into the numerical form because uh text I cannot give directly to machine learning I need a good representation so at the beginning of this uh session I told you about a representation is a very good research area so yes so the more concise your data will look like the more better the presentation the more it has been observed from the literature surveys of from the research point of view that your model can give you the better results that are making sense out of it so for this particular part I have used count vectorizer it's a basic one now we have got done I mean we have got a very Advanced embedding techniques that are being developed by Google Facebook and other ante Labs also uh for doing the same stuff so I have used here both count vectorizer and TF IDF vectorizer so uh this these both of them uh involves lots of mathematics behind it so if I try to explain you those um it will take another web series to do the same so that's why I I'm just giving you the flavor that how do they look so they basically try to do what uh they try to give the numerical values based upon the frequency frequency of words across all the documents now with the help of the SK learn live SQL library of python I have used this logistic dictation uh why process take regression here because it's a simple binary classification problem whether customer is happy or not one or zero if I would have been given a lot of data and a multi-class classification problem I would have not been chosen the logistic regression and then with the help of this uh I calculate a difference score so because of the classification stuff is there uh we doesn't we don't choose the accuracy we generally go for the F1 score so once your every uh stuff from the NLP to machine learning model is being done you need to evaluate how your approach has worked so that's that's why F1 score is there and then uh finally I have written the data to be CSV file no this is one of the visualization I wanted to show you that I got top 10 frequent words there are 100 of other visualizations like uh visualization ways uh through which you can show the various stuffs do the very good amount of Eda you can create bins histograms heat maps to uh heat maps and I would say box plots to do the same so this is what I have done so I take a pause here and again I would like to hear so was this coding part clear to you uh I'll quickly go in in the chat window just to get your responses and then we'll move to the another part yeah I have received word cloud all right yeah there are questions that I will be taking for sure at the end of the session okay I'm going to share my screen back so this was the basic modeling part that we have covered from the basic NLP stuff to a small use case uh we'll now go to the advanced one so these are just the snippet of the results that I have shown to you okay now again let's move to the NLP in the telecommunication so tele Bots uh teleport some Bots would be the new technologies that we will be seeing in the upcoming Futures so telepaths are already into the place some watts would be there so uh how do they work they get some data from ivr websites call center is call centers back office apps email chats and the social networks these teleports based upon the data and some some input from the um customers they try to create a tickets so these tickets can be an order cancellation order creation payment process ticket resolution provisioning and technical response so let me give you a more simple example uh so right now if I have to book Hospital appointments from for like hospitals quite that are quite famous in Delhi I would say um Four Tails and Max so you simply go to the website and a chatbot appears very suddenly and it asks you what are you facing where you want so it automatically creates an appointment to you and it that gets delivered into the WhatsApp so that's an example of tell Bots chatbots we can say and at the back end what is the use of this this data is being created the moment you type the data that is being shown into the one of the database files so organizations are receiving your data now no more a person calling who is not answering our phone is required so that that frustration has been decreased that a person is not transferring of calls and uh organizations also are not missing the data so it is good for both business and for us uh last and the interesting I would say also the difficult part of this session now I would be giving you a use case I will be taking you to a use case which involves the Deep learning so there was one of the uh problem that approached us from the telecommunication client only it was a telecommunication client based out of the spin and we were based out of the India and the problem was that uh their cus their business was falling off there and then uh what what was the issue they wanted to track so that they can restructure the entire internal process now they had called a manual ticketing process so what was happening uh let's say I'm calling to a helpless person and I report that so and so great near to my home is not working that's why I'm not receiving the signals like there is an indoor coverage problem or or there's a radio coverage problem if the moment I step out I don't receive a signal so all those conversations are being recorded by this uh help desk people and they are trying to uh create a ticket or even we can say based upon this conversation and uh they were trying to resolve them uh them so what was happening uh because of all this manual process the slas that is the time in which they have to solve a particular ticket was not up to the uh up to the mark So when slas are not up to the mark they cause a lot of uh business impact so they wanted to restructure this process and then they have approached us now uh challenge was uh for us was that data set was quite small and small and we need to give them the approach so that we can get a more business out of it and um then it should be it to integrate it into the form of a chatbot type apis at their level so that we they don't require more of the manual ticketing process so 86 just 86 Json files uh were delivered to them would deliver to us and it was uh the text but into the Spanish form not in in English now uh they have also shared us four manually created Excel files and the one document that was a data dictionary document so that was being uh delivered from the client side and it was restructured from our end now uh the most serious task when we talk about the industries is the data procurement and data validation so if if I get a data and I'm working as a data scientist I need to make sure that that data is enough it contains the attributes or the columns which which should be enough to solve a particular use case if it is not then uh being a techie you should go and approach to your supervisors or these stakeholders that this data is not sufficient and we should get more data that should make sense and you have to spend a lot of time in understanding that data even though you are working into the same domain because data understanding uh can give you the better connect with the data so uh just first thing is that whenever you receive the data uh create a data dictionary and uh with the help of the past experience from your supervisor or someone from operations can really help you out now uh we need to focus what are the tickets that are being reported what are the parameters we need to process like text text part description field in the data set and what are the parameters or the columns when I'm talking about the parameters these are the columns which were being shared into the CSV format that uh like the entire CSV was shared and then there were a couple of columns and based up out of the NLP approach uh once I have done the classification I need to update the columns like category criticity priority and the ticket type so all this stuff uh needs to be updated by the NLP engine so this this is the solution which we have prepared uh the very first task is that all those Latin languages uh Spanish and the other European languages we don't have sophisticated um python libraries or the r libraries to process them as well as for the low level relational languages so we do have a good result for English one but we don't have the good or the efficient one uh into for this Latin and the low level low level Indian languages I would say and uh so there were two approaches so whatever is available we should try to pick it up and then make a model like whatever few libraries are there or for or the um or the another approach would be to convert the Spanish data into the English one and then uh then do the pre-processing and bring out the result now uh when we were doing the conversion from Spanish to the Google one we can see that the context of the sentence was being broken down so what happens the moment I have converted the entire conversation from Spanish to the English one I can see now the English version but the context was broken down now what is the next step I don't know Spanish none of the team members were aware of it so we again uh interacted with some of the translator Spanish speaking person so that we can validate our data set so that our model should get maximum accuracy so um data science stuff or the LLP stuff at some point can require this uh interacting with the person who knows a particular language as well so it's it would be a teamwork not us not a single data scientist can handle it in a good way so ideally for the NLP staff if you are working in any language that is apart from the English English you you would require someone from someone from the operations to understand the data you with your technical skills as well as someone to validate your national language who is aware of that particular language in which you are working on now what happened there the data source here we can see in this architecture diagram was a trouble ticketing data so whatever tickets that have been created manually by the customers uh those are being stored here in a data source now we have showed them in a data storage proper data store and this this real-time message ingestion it it was like in the near future if they tried to give the real-time input also how this chatbots would be behaving now if I talk about just the data storage from the data storage if I pick the data then I follow these steps for the pre-processing that we have talked in the entire entire series the tokenization stopwatch removals stemming normalization so normalization uh is like bringing the data into the [Music] um into the standard format so if it is like one data is highly distributed and one data is not so uh we need to make sure the ranges are matching with each other so that it should not be here in case of the text Data ranges is like the frequency distribution so I need to make sure that those are like aligned videos then duplicate data would not be given to any of the model because it creates the Redundant information and if there is any spelling removal that also needs to be done and then manually creating problems so creating purpose means the vectorization converted into the numerical parts so once it was being done uh this NLP engine uh we did some sensor Eda because EA is the only thing we can have your trying to understand more about the data that is being offered so it's it's like your data organization is giving you data but they really don't but they have hired you because you because you I think there's some disturbance with the network thing I guess give us a second guys oh okay we need a couple of minutes to figure this out guys just a minute we are just fixing it please stay tuned we're fixing this issue so if I talk about elbows like bird RNN k n n d n n so these are quite Advanced so Transformers both model these are quite Advanced NLP models which can help me to do lots of tasks with the uh Text data so this uh specifically I have included here that uh these models are in the we can say the trend of analyzing the large amount of data as well as big data so big data is not only the large data the data which contains the higher Dimensions as well so finally with the classification report that we have created we shared them the classification that uh whatever tickets they have classified at the month and like the customer the helpless people who has created a ticket that a person is fairing a radio coverage problem or an indoor coverage or an uh outdoor published problem uh we compared that output with the output that our model gave and reported the um accuracy also we have updated the priorities which tickets should be which ticket should be more tires in the near future so this entire pipeline was delivered into the form of an API uh with with the data when we got in a large amount so we tested with the small one once it was okay we tested on a larger data and it's it's called currently also um with that organization into the form of an API so now it's like the model they try to train uh occasionally like after four months or six months I think the frequency might be this only uh they try to re-improve they try to increase or decrease the categories so that they can play around and this has uh majorly solved the problem of breaking down slas which was happening at their site so this has saved a lot of business so this is all uh the summary of the explanation that I have given to you uh here I wanted to highlight uh that we have used here lstm so uh LST model is a deep neural network based model so I would like to give an overview for this because explaining it and code for this will take another an hour to do so for this particular data session uh I will give you an overview so first let's go to the second figure so let's see this this movie is awesome so let's try to understand that whenever I get a data set like a movie review data set that textual data set gets converted into the form of a matrix this Matrix we can obtain technically in from the coding point of view by any of the embedding techniques or or maybe some Vector vectorization techniques that I have showed to you so this is required this is a representation part which is a research area that is important a step before the model and then I try to extract the important features from these Matrix and then then again again I try to keep on reducing the important features with the help of various techniques and then I give this uh feature Maps uh to the lstm model these are the different lstm units now lstm is a long short term memory model so um it was a successor of a recurrent neural network so these neural networks require RNA or lstm they have a capability of storing your long sequence of data so the traditional machine learning algorithms which can help in the classifications like naive based logistic they don't have a capability of retaining the entire sequence so there comes these type of algorithms into the picture because the context could be like he was a good player but he was he was not playing well not playing well changes the entire entire context so we need some algorithms that can uh that can create some uh they can place the probabilities or I would say um the context for the entire sentence it should it should not be biased by just the single words so that's why that's why these uh you know these algorithms are and then came into the picture number one RNN came into the picture then it had some drawbacks so we went for the lstm lstm unit if we see in the first diagram the green color one here we have a control that what to do what what information we want to give to the model and what your information uh we don't want to give so those are those are called control Gates so this is this view is something like a hardware so those people who are I mean we all have studied xor Gates nor gates in the digital electronics and then uh this this is the shape the very first X is an input input here is a embedded version that we give it to it also it meant keeps the if if it is like x x input it also uh maintains the x minus one unit this sequence and then uh as we can see here cross sign so we are just trying to control what information we required to give or what information we don't want and then from that LST model the multiple LST model we can uh do the classification so for this particular use case we did the multi-class classification model with the help of NLP and the lstn so that's all from my side so yeah thank you so much Mom yeah okay before we proceed to answer your questions I would like to request the attendees to ple