Hindi NLP - Text Generation using iNLTK & Python | Indian NLP

Key Takeaways

hey friends welcome to one little coder in this video we are going to see how you can do natural language generation that is also known as text generation for hindi language using inl tk package so inl tk stands for indic nltk so it's for indian languages which means the supported languages are hindi punjabi gujarati kannada malayalam marathi bengali tamil urdu nepali sanskrit and english so basically in ld case built on top of uh as far as i understand built on top of a fast ace library or probably just pytorch i'm not exactly very sure about it but the dependency is by torch definitely so in this video we'll learn how to do natural language generation which is uh kind of trending these days with the gpt3 stealing all the popularity so we are going to take a kaggle data set uh and then we are going to see how to do on you know get similar sentences or complete sentences based on the input sentence that you're going to give so first of all to have a look at the data set uh that we have taken so first uh thank you so much for uh gaurav aurora for making this up wonderful package that we are going to use and making it open source so the second thing is thanks to aishwarya ramchandran for making this data set available this is a data set that has got um one column that says which is the source and then the second column says what is the english sentence and the third column says what is a hindi sentence for it so my hindi my hindi is very bad so if i read something wrong in hindi please forgive me okay so first of all um the simplest thing that you can do is or to create a notebook on top of a data set go to the notebook sorry go to the dataset and click notebook so that way your dataset is already linked with your notebook so you don't have to do anything else so this is exactly what i've done so you can see after you do that thing the data set is linked with your notebook so you don't have to actually take the pain of linking that notebook or whatever you want to do so you don't have to do that so once you you're done with that thing so just execute and see if the file is in your path so it's there so our next thing is when i tried a test run with this package it actually threw a lot of warning so i don't want to spam your screen with warning so i'm just ignoring the warnings next important thing is we have to read the datasheet after you read the dataset you can probably do ahead and then try to see what is in there so you've got a source you have got english sentence and you have got a hindi sentence okay so that's a sentence uh example sentence so the thing is um on you need to have uh pytorch version one point three point one so when i tried with the latest version of pipe uh pie torch which is uh 1.5 ish so it actually threw some error so for that the developer probably has to you know re-upload the model but uh to avoid all those problems we are going to use the same thing as mentioned in the documentation which is uh python 1.3.1 so after you install that thing make sure that you have installed the library so you can use this bang operator um uh if you're familiar with any notebook setup you can actually run an environment uh system command using this operator so that's how you're able to use pip install so after you do that install the in ltk paper install nltk don't forget the bank so now in your current session so in this current session uh you've got uh the latest uh sorry you've got an older version of pytorch which is 1.3.1 which is cpu version uh i think there is no cuda support at this moment and then the second thing is you have got nldk in ltk library so now that we are done so the first step is you have to load the pre-trained model so load retrained modeler you can call it language setup whatever you would like to do you can call it language setup okay so the first thing you have to do is uh load uh setup from in ltk and then what is the language that you're going to set up so you can see here for the language that you are going to use you have a code ha is for a hindi and um tas for tamira whatever it is right so you can use it uh so it throws an error um i don't know why i'm not raising an issue for it but you can see that it is done so this is done so uh we just saw a head so we'll pick one particular sentence in this case the sentence that is slightly easier for me to read um so this is a sentence that is easier to read for me uh so it uh translates so i would like to tell you about one child on it so that is what we are going to look at so we will load that sentence and then we are going to first tokenize as in every nlp you first start with tokenizing um but ideally in this tutorial we are not going to do anything with tokenizing at all uh it is not required but i just wanted to show you how it tokenizes uh so you tokenize it you can see my is like this i'm not really good with hindi grammar for me to say that it is right or wrong but this is the case um it looks good to me um but you you can like if you know hindi and you're watching this video please point it over if it's not doing a good job so what we are here is of our generating text a new text okay so the text that we are giving uh has to be used and then some new text has to be generated so the first there are two types of text generation we are going to do one is we are going to pass on this text and then we are going to ask this package the function gate simulation tense to create new text for a similar text similar sentence so let's call it a similar uh sentence generation okay so what we are going to do is we are going to import the function get similar sentences and then we are going to pass on the text that we have which is that i would like to tell about one child and then we are going to say okay what is the degree of argumentation you want to change based on that uh you can play with it so and you can say okay how many ticks you want so let's say i want to change point three and then how many texts i want so like i said this is all the warnings that gets uh shown um so the three sentences have been generated for our sentence which is i'll read the third one which seems uh okay i'll read the second one okay so i think it is it is replaced rather than testing my hindi skills translate okay so it says i would like to see uh you about i think it is probably not doing a great job maybe because the argumentation is too much so let's try point one and then see what is it um let's see yeah it is generated so i'll just translate it right away so it says i would like to tell you more about such a child okay that is good that is promising let's take the other one on july and let us see july would like to tell you about one child so okay uh not bad not not a bad thing and i would like to tell you about uh those one child so it does its job so the point is uh you gave a text and then you got another bunch of similar sentences so you've done some kind of natural language generation here like let's say if you want to run a twitter bot um who cares what is the grammar so people actually need to some something that makes sense to them so this idea looks like it makes sense to them so one of the things that with um gpd3 if you have seen gpt three demos gbt3 basically are any likely works based on a prompt uh so you have to give a prompt and uh that prompt based on that prompt next words get predicted so this is exactly what we are going to do so we are going to give a prompt which is the sentence text uh text generation based on previous words so what this is going to do is it's going to take these words what we have already given that uh i would like to tell you about a guy boy and then we are asking this um package uh using this function predict underscore next underscore words to predict uh some other next you know sentences or words um so number of words you want let's say we want 10 um that seems ideal so i again do not want to risk my hindi skills so i would like to tell you about one child who matches india and chemical fertilizers um now you know probably why gbt3 is really popular um because of its accuracy but uh as you can see we are just running this on a cpu we have not done any um our own text documentation we have not any we have not done anything at all just we have taken an out of work solution then we have tried to build something on top of it and for that i think it actually works good uh so let's try another sentence it says i would like to tell you about one child in which the years the sentence look decent so let's generate okay if you want reproducibility you have to definitely set a seed value i'm not doing that here just because i'm using it for a tutorial purpose but uh if you want uh you can do seeding this okay so looks like the sentence itself is bad so the basic ideas i think you have got uh so this package is a really really good package if you want to play with indian languages and you can also you can also have a look at uh where is that you can also have a look at the embedding um using a tensorboard uh you would have a link here um based on that you will have understanding about how the the language model is that what kind of words look similar closer to it so you can have a look at that so at this point on this uh using this package just out of box solution without doing anything other than a couple of lines you can generate your own hindi text based on the text that you have given it could be either the similar sentence or it could be based on the previous text as a prompt or you can generate a new text couple of words that you would like to predict so this is a very handy package if you work in an environment where you need to do something in indian languages or maybe you want to create a twitter or a reddit what what kind of or you want to just send a whatsapp message using selenium whatever it is this is a very good package and once again thanks to garrow for making this package and uh of course as you know we will link this uh this is a public notebook this is going to be public notebook oh it's already a public notebook so i i'll link this just below the like button if you enjoy the video please like the video and please share it with your friends hindi friends and uh if you have any comments or suggestions about my hindi skills please let me know in the video comment section thank you so much for watching this video see you next time