Text Generation AI - Next Word Prediction in Python

Key Takeaways

what is going on guys welcome back in this video today we're going to use tensorflow and neural networks to predict the next word of a given text sequence and we're going to use that mechanism to generate text with this AI so let us get right into it [Music] now before we get to the actual video I would like to mention that this video was sponsored by bixley and don't skip this part because this could be quite interesting to some of you guys working on software projects oftentimes software projects are quite difficult to finish and Bixby can help you with that they have been successfully completing projects on time and on budget since the year 2008 so they have a solid track record of successful completion with the money they have with the time they have and they offer multiple different Services you can go to their website you will find a link in the description down below bixley.com and N for neural nine and you can click on their services tab for example to see what they offer here they have different services like no or low code type of services Enterprise development devops mobile development and more you can take a look at that and if you need some help getting your project finished on time or getting it done right the first time you can talk to the people at bixby.com as I mentioned you will find a link in description down below and they have a solid track record in addition to that you can also get a free consultation by visiting their website so on the link in description down below alright so we're going to build a text generating AI in this video today and to be precise we're going to build a neural network that always predicts the next word given a certain text sequence so for example we might have 10 input words uh the start of a sentence and our model will always predict then the 11th word or maybe it will give 10 suggestions for the most likely next word and this can be used first of all of course just for the next word prediction for auto completion or something and it can also be used somewhat at least for text generation we can attempt to do text generation with that because the act of text generation is just always predicting the next word and the next word and the next word and the next word uh there's not much more to it this is what we're going to do in this video today we're going to do that with tensorflow we're going to build our neural network with tensorflow in addition to tensorflow we're also going to use numpy pandas and the natural language toolkit and ltk those are the libraries that we need to install and we can do that by opening up the command line and typing pip install tensorflow numpy pandas um and nltk and we're going to need nltk because of course in order to train our model we need to work with some Text data so we need to have some raw text and then we need to tokenize it and stuff like that which is why we need the natural language toolkit and the data set that we're going to use in this video today is going to be the fake news data set that you will find at the link in description down below you just go to this GitHub repository here you click on the data directory and here you have this fake or real news CSV file this is just a simple data set containing news articles some of them are fake some of them are not fake but at the end of the day all of them should have coherent sentences so we can use them to predict next words and you will see that the data set you use is going to be influencing the text generation so if you use this data set you will get a lot of information about presidents and some political stuff whereas if you use a Shakespeare data set you're going to get some more poetic text um but yeah you need some data set you don't have to choose this one all you need is you need to have a string in Python you can store this string in a variable text so just have something text equals and then you need to have a lot of sentences here so you should have like a hundred thousand characters one million characters whatever um we have sentences so that you can see a lot of examples where you have one word following the previous words this is what you need we're going to get this from the fake or real news data set but let's start with the Imports now we're going to import random we're going to import pickle we're going to import numpy SNP we're going to import pandas as PD we're going to import matplotlib actually we don't need map.lip we need nltk dot tokenize or actually from nltk dot tokenize we're going to import a regular expression tokenizer so reg X Rec X tokenizer we're then going to import uh tensorflow or actually let's do it with the from imports from tensorflow from tensorflow Keras models we're going to import the sequential model and we also want to import the load model function we also want to import from tensorflow uh Keras layers if you want to import lstm dense layer so lstm is basically a recurrent neural network layer a long short short term memory and this is what you want to use for stuff like next character prediction or even stock prediction stock price prediction or next word prediction this is a very useful layer for this uh the density is just a fully connected basic layer and then we also want to have an activation layer and then we want to also say from tensorflow dot keras.optimizers we want to use the RMS prop optimizer those are now the inputs for the text it's quite simple we're just going to load the text Data frame uh using pandas so read CSV we're going to load this fake or real news CSV file and we can then print this and you can see that it's a basic data frame where we have the title The text the actual news article which is what we're interested in in the label now since we're not going to do a fake news detection we might do that in a in another video soon uh but we're going to just focus here on the raw text we don't care about any machine learning prediction task here other than the next word prediction we don't want to know if that's fake or not we just want to get the text and predict the words in the text so what we're going to do now is we're going to say that the text is going to be a list of the text DF dot text value so we just take all these texts uh the list basically we turn this into a list and what we do then is we say that joint text is going to be white space on this white space character join all these texts and now I'm not going to print that because that's going to crash The Notebook because it's just too much text but this is now just raw text all these articles um one after the other that's the basic thing that we have here um and what we can do now to not uh you know train them all for too long before it actually works let's just go ahead and train this on or train them all on a part of the text oftentimes you will not be able to do it on all the text uh anyways because we're going to you're not going to have enough RAM to store all of this so what we want to do is we want to say partial text is going to be equal to Joint text and we're going to say up until um 10 000 characters just for for testing purposes here now what we want to do with all of this text is we want to tokenize it so we want to say that the tokenizer is going to be a regular expression tokenizer and the regular expression is going to be quite simple just words just tokenize the individual words and the tokens are then going to be tokenizer tokenize and we're going to just tokenize the partial text now to keep things simple to not have too many different words and to also be a little bit tolerant with different cases we're just going to say here um that we want to tokenize only lowercase so we're not going to care about um uppercase or lowercase characters here everything is going to be a lowercase and the generated text is also going to be lowercase so those are the tokens now we can print the tokens probably there you go there's just a list of the individual words that we have here and what we want to do here is sorry what we want to do here is we want to say unique tokens are going to be equal to numpy Unique so that we don't have duplicates tokens and what we want to do now is we want to have a dictionary so that we can match a token to a respective index so when we have a word we want to know what's the index in that list and for this we're going to say unique token index is going to be a dictionary where a token is mapped to an index for index token in enumerates unique tokens that's what we want to do here so maybe we can look at that there you go each word that we have is going to be mapped to a certain index all right so that's what we have here as a basis now what we want to provide here is a parameter or at least we want to know this information you don't have to store it in a variable but we want to have uh some number of words that we want to look at so how many words do we want to look at in order to predict the next word do we just want to look at the last word and then predict the next word this is not very useful because you know you don't get any context if you look at the last 1000 words maybe you get too much context so let's just go with something like 10 let's say n words is going to be equal to 10 and then we're going to have a list input words and we're going to have a list next words and this is going to be just a structure for X and Y X is going to be the input words Y is going to be the next words so for each position in our text in our joint text here what we want to do is we want to get 10 words we're going to put those 10 into input words we're going to get the 11th word which is going to be the next word and then we shift by one so we're going to get the next 10 words so all except for the last one is going to be um are going to be the same in different positions of course and the next word is going to then be the 11th word after that sequence and we're going to do that for each position in our text and this is going to result in our training data so we're going to just do it like that and we're going to say here for I in range length tokens minus n words now we're not going to iterate over the joint text we're going to iterate over the tokens because here we we tokenize for a reason because we then have similar words grouped into a token um and we say here minus N word because of course we don't want to go until the end because then we don't have any words to predict anymore uh four range length tokens minus n words what we want to do is want to say input words a pent and wanna append all the tokens from the respective index starting at zero up until that index plus n words so we're just looking at the 10 words after the current position and the next word is going to then be the 11th word so it's going to be tokens I plus n words there you go and this needs to be plural there you go next words is always one word and input words is always a collection of 10 words so we can see here Daniel Greenfield uh a showman journalism fellow at the Freedom Center and then the next word would be is right and then is a New York writer focusing on reticle and so on that's the basic idea of what we're going to do here um and now we want to turn all this into X and Y so we're going to say x is going to be a numpy array full of zeros with the following structure length of input words n words and length of unique tokens the basic idea here is that we wanna um for each sample we want to have n input words and then the actual um then a Boolean for each uh possible next word so I want to have um basically for each sample so we have input words the length of input words amount of samples then we have the 10 input words and then we have for each token that we have for each unique unique token we have either zero or one if it's the next word or not so we have this bag of word structure where we have for each token uh a binary feature and all of them are going to be zero except for one it's going to be one the next word is going to be one and this is how we're going to pass this to the neural network and the data type for this is going to be a Boolean that's that's then for the Y we're going to do something similar we're going to say NP zeros but this time we're going to say length next word basically should be the same um length unique tokens because here we don't need input words so what we do here is for each sample we just do the same thing what's the next input word uh sorry my mistake here we have only one set to one here we have up to 10 set to one up to n words set to one because um what we have here is We have basically the input to the model so what are the tokens that are activated and then we have um let me let me rephrase that what we actually have in the X array here is we have for each sample so for each selection we have n different so 10 in this case 10 different bag of words where only one word is set to one so you have basically 10 words but you don't have 10 words of strings you have 10 words represented as a bag of words where you have just one index one position being set to one all the other to zero but you have the ten times because you have 10 different words and here you just have this one one time because you have just one next word to predict I hope this was not too confusing I made some mistakes in the explanation here um here of course we also need D type equals bull but what's the problem here module numpy zeros there you go next word is not Define why do I keep using this form it's plural [Music] um there you go so X now looks like this false false false false false and the the word that is used will be set to true all the other fields are going to be equal to false and Y looks like this you have the same structure but you only have it once because you only have one single word um that's the basic idea now what we want to do here is we want to say four index and words in enumerate input words we want to say also for J word in words or enumerate words we want to say that because this here what we have here is just a structure and now we want to fill up that structure with the actual values so we want to take the position the current sample the current word and we want to get the word at this position and map it to the respective index which is done by the dictionary that we created and want to set that position by one so again let me put this into somewhat simple term terms we have the structure that I just explained and what we do now is we we go through all the samples that we have in input words we go through all the words per sample and then we go in the X array which is our input to the neural network we go to the sample position in that sample position we go to the word uh that we're currently looking at and then we take that word map it to the respective index find that index inside of the array that we created and set only this position to one or true if you want doesn't matter and for y we do something similar but we skip the J part because we only have one word um and here we need to also use next words I because that is um that is what we're looking at we're not looking at uh this word but at the next word all right so and of course this obviously is not done inside of the loop this is done only once and that should be it let's see now we're not going to see the results probably because we only see the false instances here but now we have the values actually in in our arrays in our X and Y arrays uh so that's basically it now we can go into training the model and for this we're going to have a very simple structure we're going to the same model equal sequential model we're going to say model at simple lstm layer 128 neurons the input shape is going to be n words depending on what this variable is then we're going to have the length of the unique tokens um and we're going to say return sequences equals true because what I want to do next is I want to add another lstm layer and if you have multiple of those you need to add this return sequences to all but the last layer so since I'm going to have two I'm just going to say lstm 128 here we don't need to return sequences here and then we want to say model at dense layer and we're going to say length unique tokens here and we're going to say finally we want to have an activation layer with the softmax function so each word will have a certain probability which also makes it easy for us to see okay if we don't want to take the best option or the most likely option maybe the second most likely option is the right result so we can filter for the N best Solutions the N best predictions that is the model input shape is not defined uh uh what did I do wrong here input shape is not defined oh because it's not a function it's a parameter there you go model dot compile we're going to compile using the categorical cross entropy since we are predicting using the softmax function here multiple categories the optimizer is going to be equal to an RMS Prime Optimizer with a learning rate of 0.01 and the metrics that we want to look at are just the accuracy and then what we can do is we can store actually we don't need to store the history we can just say model dot fit X and Y and then the batch size is going to be 128 we're going to train for 10 epochs you can also train for more than 10 epochs and we're going to shuffle we're going to set shuffle to true um and then we can fit the model if everything was done correctly and you can see since we just used 10 000 characters this uh is quite fast we can also set this now to 30 epochs maybe and we will get some decent accuracy on on the test set or on the training set here but that doesn't mean that our prediction our text generation is going to be perfect just because the accuracy is high here just means that it's pretty good at doing this on the text that it was given now we're not gonna care about training the perfect model here let's just see how we would produce the results before we go into more detail on a better model or anything let's just say this is a good model how would we then use it to predict the next word and how would we use it to generate text now first of all we want to save the model so we want to save the model here into uh my model.h5 and we can also load them all by saying model equals load model and then my model H5 we don't have to do this because we still have the model in our Ram but if we want to load them all we can just do it like that and then we have the same model now uh with this model we can predict the next word by defining a simple function predict next word and we're going to have the input text here and we're going to have the N best parameter meaning we want to have the N best the end most likely predictions so what we want to do here first is want to take the input text and turn it into a lowercase version of itself so that we don't have any uppercase characters because we don't have um we didn't train them all on uppercase characters so it might confuse it or it will confuse it we take the input text we turn it into lowercase and then what we do is we say the x that we want to feed into the model is going to be an array full of zeros with the following shape one because we just have one input text n words and length of unique token so basically the exact same um the exact same shape as up here but now we only have one input sentence or 10 input words you could say so we only have one here um and what we do now is we say for I word in enumerate and we want to split the input text um on white spaces for each word in this text what we want to do is we want to take this word and want to see if we can find it uh in our unique tokens list so what we want to do is we want to say um x 0 I unique token index for this particular word equals one and this of course can produce an exception so if we don't find that word in our dictionary this will basically tell us that there is a key error we didn't find that word so we cannot set this position to one and it will not predict the next words how you handle that you can think about this so you can just in in um in the case of this problem you can just say Okay predict any word or give me a random word or maybe as an input use a different word what we're going to do is we're going to just accept then in this case it doesn't work and in the text generation we're going to just go with a random Choice with a random word instead of the most likely word um but of course there are more intelligent solutions to this so the predictions are then going to be um so this basically just creates again the bag of words with respective ones in the positions um and then what we want to do is want to say model predict X and we want to get the result so index 0 here those are the predictions and what we want to do now is we want to return the N best predictions and we can do that by using a numpy function called Arc partition so NP Arc partition we passed the predictions we pass negative and best and we get everything from negative and best until the end that is our function and this should now already work so if I say possible um is going to be equal to predict next word and I can provide a sequence like I will have to look into this thing because I those are pretty basic words I hope all of them occur and let's go with the five best predictions uh key error I okay it doesn't know let's go with he maybe because we train this on okay it doesn't know because since maybe since it's also not known let's go with uh this thing and he this should work and now we can get the possible next words now here we get an array with indices what you need to do in order to get the actual words quite simple you need to say uh print and then um unique token tokens index for index in possible and of course we need some parentheses here we just have to take the index and get the respective token from the list in this case those are the five most likely words he will have to look into this because he appearing he mate he public he at he it those are garbage predictions because the mall was trained on 10 000 characters this is not much at all but we can see that it makes predictions and the more data you train them all on and the more complex them all is the the better them all is the more accurate the predictions are going to be here now before we look at a model that I trained already for quite some time let's look at how we can use this now to generate text now you can see since this prediction here is garbage the text generation is also going to to be garbage but let's say it wasn't as bad as it is right now how would we take this and generate text quite simple we we Define a function generate text and we provide an input text that has to have the length 10. we provide n words and we provide I mean actually we can take n words from above already so let's maybe not pass this and we provide a creativity and the creativity is going to be set by set to three this is just going to be uh how many words do we want to choose from we're going to make a random Choice uh what the next word is going to be from the N best choices what is this n in this case we're going to look at the three most likely choices and um and basically uh and and and choose from those now we are going to use a parameter here though we're going to call it text length uh because we need to also specify how many words we want to generate so we're going to say here that a word sequence is going to be input text split we're going to split it on the white spaces so that we get a list of of words and we're going to then say the current position is zero and now for placeholder in range text length so for how many words we want to generate we're going to say subsequence equals white space join tokenizer tokenize join the word sequence that we currently have in lower case obviously and we want to take the current position up until current Plus uh now this is now plus n words right I think so should should work um so this basically takes a subsequence and now I would want to do is want to try I want to try to um find the next word we want to try to predict the next word and this will not work of course if um we have some unknown input so what we can do here is we can say Choice equals unique tokens random choice and we're going to predict the next word from the subsequence that we're looking at so from the last 10 words with the creativity that we're given here so the N best words we're going to choose from this list and we're going to take that respective token if some error occurs for example because we cannot find this respective T we're just going to pick a random Choice from the list of all tokens so Random Choice all tokens that's what we want to do here and then whatever the choice is and whatever choice or however the choice was made we're going to just append this choice to the word sequence and we're going to shift the current index by one and then finally once we're done with this loop we're going to join all of this on white spaces and we're going to have some text that is going to not make a lot of sense but it is going to be text so let's go ahead now copy this down here and let's say generate text this is the input 100 is the length and 5 is the creativity let's see what happens he will have to look into this thing and he public it who's KGB if woman political head whatever right so we get this text that doesn't make a lot of sense you can already see that this is very political uh and stuff like that so this is because it was trained on a news data set fake news and real news data set um but this is how you can do this how you can do this here now let's go ahead and use a model that I already trained on this data set for quite some time I think I used a hundred thousand characters so instead of using 10 000 here I used either a hundred thousand or maybe even I used I'm not sure if I didn't use a million but I trained this and the results are not that bad so let's see if we can get some better generated text here by saying model equals load model and I stored it here in the pre-trained directory so pre-trained model.h5 or actually I think it is pre-trained models model H5 and I think then I should be able to just use these functions again let's see what we get here um incompatible why is that oh because I need to also load the same amount of data here so let's see maybe I have to go through all of this again but not the training part and then no we get some problem here incompatible probably because I still didn't use enough so let me add another zero let's go ahead and do this again there you go now it does something and hopefully they go he will have to look into this because he makes he could he has he is he basically also he is those are all reasonable predictions so at least if you want to have something for auto completion if you want to have something for the next word prediction this is actually decent I still think that generated text will not make a lot of sense but let's see what happens if we try to generate text with this he will have to look into this thing and he makes a replacement in our history so even though it does make a lot of sense what we write here it is more more coherent the words St you know the words fit together at least somewhat um so this is actually kind of cool let's go ahead and try something else like I want to see what is possible uh one two three four five six seven in this world are you it we is so maybe let's go the presidents of the United States has announced one two three four five six seven eight that he will he had he's he has he is those are decent predictions so for the next word prediction let's go maybe with 10 to see what we get here he would he said he could he he doesn't he is he was he has he had he will all of those are decent predictions that is really really good [Music] um let's go ahead and see what kind of text it would generate if I provide that and in this case we get the president of the United States has announced that he will be more stressed of that is a former this we will is president president yeah I mean you can see it's better definitely but it still produces nonsense let's see how this changes if I just predict the best word every time in this case we would get only the best the most likely solution the president of the United States has States has announced that he is a good president of their children when the dreams application of the president when the president's sexually on the okay um and then maybe if we go with a creativity of 20 this will produce probably some garbage but let's see the president of the United States has announced that he did a president before his own the election he took out and so on uh so yeah the text generation part is not too impressive but the next word prediction part is pretty impressive and of course if you're training this on more and more texts on different data sets with a larger model and you do this with maybe multiple days of training on a high-speed computer this is going to give you some decent results over time and uh yeah this is how you can do that with tensorflow with neural networks in Python so that's it for today's video I hope you enjoyed it and hope you learned something if so let me know by hitting a like button and leaving a comment in the comment section down below also don't forget to check out Bixby the sponsor of this video you will find a link in the description down below to check them out and as always subscribe to this Channel and hit the notification Bell to not miss a single future video for free other than that thank you much for watching see you next video and bye thank you [Music]