Generating Poetic Texts with Recurrent Neural Networks in Python

Key Takeaways

all right there we go we have the results so with the temperature of 0.2 we got something like Camillo I shall be the seams and the shall be the death that the will the will and the will the world so yeah what is going on guys welcome to this video today we're going to start with our first AI project so we're going to complete an AI project in Python and this project we'll be building a recurrent neural network that generates texts like Shakespeare's texts so of course it is not going to be the same quality but basically what we're going to do is we're going to take Shakespeare's text feed them into a recurrent neural network this neural network then learns to produce text based on that and in the end we'll end up with a neural network that can produce and generate its own texts based on Shakespeare's text so there will be kind of similar now before we get into the code and into the implementation I just want to mention that our neural 9.com you can find a text-based version of this tutorial so basically you can find also a link in the description this is essentially the same content that we're going to cover today but if you want a text-based version of this just check out the link in the description or go to neural nine.com and look through the blogpost basically this what we're going to do today so let us get into the implementation now let us start by talking about the libraries that we're going to need today now first of all we're going to need the random library of the basic Python stack for a helper function later on of course we're going to need numpy as NP we're going to need tensorflow because this is like the basis of our script this is all the neural network stuff we'll be done in tensorflow stf so we import the essential tensorflow library but we're also going to import some specific things we're going to import from tensorflow dot Chara's dot models we're going to import sequential so that we can create the neural network later on we're going to import some layers so tensorflow carest layers we're going to import arm the LS TM layer which is long short term memory which is basically a recurrent lair with a memory you could say so this will be the memory of our script of our model sorry then we also need dense layers for the hidden layers and an activation layer for the output layer and we're going to choose an optimizer because later on we need to compile them all so we're going to choose the rmsprop optimizer these are the imports that we're going to need now if you're confused and you lack basic machine learning knowledge you don't know what numpy is you don't know what tensorflow is check out my tutorial series on data science and machine learning that should clarify the basics and I'm not going to go to explain to explain all the basics in this video so check out the tutorial series for that now the next step is getting the file that we want to train with so basically we need some Shakespeare texts are in order to to have something that our model can be inspired by quote-unquote now I have a link for this which comes from the official tensor flow cameras tutorial so this link you should find it in the description if I didn't forget to add it and basically when we enter it in our browser you can see that this is a decent amount of Shakespeare text goes on for a while and we're not going to download this file manually and read it into Python but we're going to use tensorflow to directly loaded interest script so we're going to say okay file path equals and now we're going to say TF Chara's dot util dot get file and we're going to specify a local file name let's say Shakespeare dot txt and then we're going to specify the link which is this one as I said if I don't forget it you will find this link in the description otherwise you just have to type it yourself so the next step of course now that we have the file is we need to get this file the text of this file into our script so what we're going to do is we're going to say text equals and we're going to open this file so the file path here disfunction get file returns to file path into this variable so we're going to just say open file path in our read binary mode so yeah basically just read binary and then we're going to say dot read so that we get the text but we have to decode it so we're going to say decode the encoding is if I'm not mistaken utf-8 so this is the encoding and what I'm also going to do is I'm going to transform it into a lowercase because later on we will have some possibilities for a next character our model will always predict the next character and if we also allow for uppercase characters it has way more possible choices so the accuracy is not going to be that great so we can increase the performance by only using lowercase letters now of course for the grammar and all of that it's probably not good but for the content for the semantics it doesn't make a difference so we're going to use lower case characters only so that we get a better performance in the end and that's basically it now we have the text in our script now the next step is to somehow get this text and convert it into a numerical format because we're a neural network will not be able to deal with sentences we have to convert these sentences into a numerical format so that we can pass a numpy array into our network we cannot just pass some sentences and say predict the next character or something like that so what we're going to do is we're going to find a way to convert the text or each character into a unique numerical representation and then this numerical representation back into the same character we'll do this with dictionaries but before we do that we're just going to select one part of the text because training our the neural network on the whole data set on the whole text might take a little bit too much or too long so if you have time or if you have a very fast computer you might also train it on the whole text but for this video I'm going to select some part of the text so let's say from character a 300,000 up until character eight hundred thousand these characters will be the characters that we're going to use so the only thing that we're doing is we're just selecting one part of the text instead of the whole text now what we need to do next is we need to create a character's MMA character set which basically contains all the possible characters all the characters that occur somewhere in the text so for example if the whole text doesn't contain a single T character we're not going to find T in this data set of course all the characters are appear somewhere in the text but just theoretically if a character doesn't appear anywhere in this section that we selected it will not be in this set so we're just going to say characters equals and now we're going to say sorted set off text now the set function what it does is it filters out all the unique characters so it says basically if a if an X appears that's X 1 X it's not all the axis in a set it's just every character occurs only one time and sort it is basically just sorting them so if we have ABCD in the text somewhere or let's say in a different in a different order let's say we have D F and then a and G and B in the end by using the sort of function we're going to have ABCD and they're going to be enumerated in the end with 0 up until the max or up until the amount of characters that we have for possible characters that we have so this is just a set of all the characters that we can have which is sorted that's basically it and now what we need to do is we need to create two dictionaries so that we can convert these characters into numerical format and the numerical format into the characters back into the characters so what we're going to do is we're going to say character or let's say dress char to index equals and now we're going to create a dictionary and this dictionary will contain C and I which stands for character and index and for every I and C in enumerate characters so what we're basically doing here is we're saying create a dictionary which has the character as a key the indexes of value so basically we find the index we find a number by passing the character for all the indices indices and characters in the enumeration of characters so the enumerate function what it does is basically assigns one number to each character in this set and what we have here is basically something that looks a little bit like that you have one character for example a and this a refers to a one and we have a character F and this character F refers to 27 or something and we have a dictionary with all the possible characters that we can have enter the respective number or numerical representation and of course we're going to do the same thing let me just copy this line here we're going to do the same thing the other way around which will be called index to char and the only thing that we need to do is to swap the swap these values so basically now we have the opposite we have the number as a key and the character as a value so we'd refer to the number and get the characters so this would look like one would be a or 27 would be a half and so on and this is what we're going to use to convert our text into numerical format now we get to some interesting part that might seem a little bit confusing but it's actually quite simple what we now need to do is we are going to create a oh sorry I've caps lock we are going to create an empty list of sentences and an empty list of next characters the next characters will be the target and the sentences sentences sorry will be on the features so we have a sentence we scan a sentence we load a sentence into our neural network and the result will be the following character so for example if the sentence looks somewhat like how are you next character would be a you because this would complete the sentence and what we want to do is we want to feed a bunch of characters let's say 40 characters into a neural network and then as a result we want to get the next character the 40 first character and what we need to specify here is how long the sentences shall be so how many letters or how many characters are we going to use as features or as feature data in order to predict the next character you can say predict the next character based on the last five characters the last 500 characters doesn't matter of course you need to be careful because you don't wanna you don't want your network to rely on too much data but also if you just choose two characters you will not end up with a reasonable text so what we're going to do is we're going to define a sequence length of 40 which means that we're going to use 40 characters in order to predict the next character and we're going to define a step size and the step size is basically we're going to choose a step size of three the step size is basically how many characters are we going to shift to the next sentence so if we have a hello world I love neural nine because the videos are so great if you would have a sequence length of five this would be our first sequence now the step size three means that for the next sequence I need to shift three characters which means that the next sequence will be this year so we shift the first three characters the beginning shifts three characters and then we start with the next sequence so of course we have an overlap here but basically this is what we're doing we're choosing a sequence shifting X amount of characters choosing the next sequence shifting again and so on until the whole text is in our list because of course this is a training data now what we need to do is we need to fill up that list based on the sequence length and the step size so what we're going to do is we're going to say for I in range I just need to look a little bit on the cohere because it seems kind of confusing from in the range 0 to the length of the text so as long as the length of the text is - the sequence length because of course we're not going to start a sequence in the end we need the sequence length to be there and the step size is of course the step size so this loop basically runs from the beginning of the text until the last part of the text where we can read a whole sequence length and always uses a step size of 3 in between and we're going to now do is we're going to say sentences dot append so we're going to add the sentence I up until i + sequence length so I is basically the position that we're at right now the starting position that we're at right now in the sequence length is basically just the length of the sentence so I until I + sequence length up sequence length sorry will be the sequence that we want to add to the sentences and this is basically it so we have from I actually sorry I forgot to specify the text of course this is just the index so we're going to say sentences append the part of the text that reaches from I up until I plus sequence length and in the second step we're going to say next characters dot append and we're going to use text I plus sequence links so basically what we're doing here is we're saying if the sequence length is 5 we're getting the character 0 up until 4 and then the character with the index 5 is the next character which is of course what we want to have so we want to have training examples where we have a bunch of sentences and always the next correct letter and this is how we do that and this is how we fill up the two lists so now we have the future data and the target data but it's still in a string format in a character for me we need a numerical format and therefore we now have to convert the training data that we just created into a numpy or into two my array so we're going to say x equals NP zeroes so basically we're creating a numpy array full of zeros and the shape of this shall be the length of the sentences so basically how many sentences there are which is not the length of the sentence because this would be the sequence length but the amount of sentences we have times the sequence length so the length of these sentences times are the amount of possible characters and plot twist the data type of this will be boolean so NP dot and P dot o and how this works is basically we have one dimension for all the possible sentences we have then one dimension for all the individual positions in these sentences and one more position for all the possible characters that we can have and what we basically do is whenever in a specific sentence at a specific position a certain character occurs we're going to set that to true or 1 and all the other values will remain 0 basically so for example if we have sentence number 5 and at the position number 7 we have the character with the enumeration or enumeration 8 we're going to say X 5 7 8 I hope these numbers were the same I forgot them 5 7 8 equals 1 because there a character occurs and in this format we're going to pass a training data to our neural network and the output the target data will also be in a similar format but not the same format it will just be a length of sentences so basically amount of sentences times length of characters and we'll just say which is the character R which would be the next character for which sentence so we're saying at sentence number five the next character is the character with the enumeration eight and then with no case sentence number five the next character of that is eight which is for example I don't know a P or something like that and of course the data type of this one is also an P dot boo that's basically it now we just have to fill up these arrays so now we have to fill up this erase with two for loops so we're going to say for I and sentence in enumerate sentences we're going to say for T and character in enumerate sentence not sentences sentence we're going to say X I T and character to index character actually I think I need to use square brackets here is going to be set to one and we're also going to say okay why I and character to index next character next character or next yeah next characters I equals one so what are we basically doing here we're running one full loop over all these sentences so basically we're taking all the sentences that we just created all the sequences and assign an index to them so zero sentence first sentence second sentence and so on and for each of those sentences we then again enumerate every single character in the sentence so we say R T which is the character of the sentence or the position of the sentence character we say basically index that as well and then we say the the position in the X rated we just talked about with the boolean sentence number I at position number T and character number whatever so we take the character datas at this position convert it into a numerical format this position is set one because that character occurs at that position in that sentence therefore it is true one and this is the training data for each single sentence for each position in the sentence and for each character we have either a 0 or a 1 and in the output in the target data in the y array we have the same thing with eye and character to index so we basically say ok this sentence the next character here is this one and this is what we do with these two for loops so now we get into the essential part of this video which is building the neural network now we have prepared the data we have perfect training data which is split up into two non point arrays x-ray and Y array and now we're going to do is we're going to take these two arrays feed them into a recurrent neural network so that this neural network is trained to then perform the the job of predicting the next character so what we're going to do is we're going to say Mall equals TF erection of TF sorry sequential because we imported it directly mall equals sequential and the first layer will immediately be a LS TN layer so we're going to say mall not ad LS TM and lsdm as I said stands for long shirt short term memory sorry and it is the memory of our network it will remember the past couple of characters the important characters they're relevant characters because if we just feed the data through it it will always look at the input that it gets right now and not remember the input that it got a few iterations ago so we're going to define it with 128 neurons and an input shape of input shape that was written input shape equals the sequence length not sequential sequence length times the length of characters so that is the input shape of our LS TM we get the inputs and immediately put them into the lsdm layer this layer of course is followed by a dense layer which is just adding the complexity to or mauled a hidden layer here and this dense layer has as many neurons as we have possible characters so if we have 10 characters in all of our text this will be a layer with 10 neurons but we're going to have a couple of more and in the end we're going to wrap everything up with an activation layer so we're going to seem all that add activation and this activation will be the softmax activation function now if you don't know what saath makes thus I've talked about this in my machine learning tutorial basically it scales the outputs so that all the values add up to 1 this basically means that the output is always a probability of how likely a certain character is going to be the next character so for example if I have a base sequence as an input I can have multiple characters with a different likelihood for example I can say the next character character is 70% a K but it could also be for 20% and X or something like that so this is what softmax does it basically scales the values so that they add up to one or a hundred percent basically now what we're going to do is we're going to compile them all we're going to say model dot compiled and we're going to define the last function to be the categorical category categorical cross entropy and the optimizer is going to be rmsprop with learning rate of learning rate of 0.01 and of course we're going to just say mauled fit and we're going to fit it on the training data so X Y we're going to choose a batch size here which means how many it's a little bit laggy right now I don't know why which means basically how many examples are we going to put into the network at once I'm going to choose 256 which is a power of 2 and then we're also going to choose epochs which is how many times our network is going to see are the same data over and over again so this is how we train them all now we need to do one more thing before we start the script we need to fix a mistake I made up here when creating the arrays X&Y what we need to do is we need to surround the shape values with parentheses because it has to be the form of a tuple the shapes are always a tuple so you cannot just pass the values here you have to pass a tuple of the values because otherwise you get the error that you have too many arguments too many parameters for this function and also we're going to add one more line at the bottom which says small dots safe text generator or something like that mom and this is important because training them all will take quite some time and we're not going to do it over and over again every time we run the script so we're going to train at once then save them all and later on load them all instead of training it over and over again however I'm not going to run the training arm on camera here because as soon as I click on run the whole recording lacks and you cannot even see anything off the screen anymore because it's completely laggy I don't know why my computer is actually not that bad maybe I'm just a noob at recording videos but I'm going to run the code and then come back to you once it's trained now as you can see the model is already trained and it ended up with a loss of around 1.5 for and we also have them all saved here in our file in our folder sorry so this means that we can actually get rid of all of that here and just replace it with ma equals T f dot Kira's euros of not you oh sorry that models dot load model and here we can just load text generator dot model which is a file which contains remote so instead of training it over and over again we can just load it into our script now actually I think we could also get rid of all that here but I'm going to ignore it because I'm not 100% sure but actually this is just a preparation for the training data and we're not going to use it again so maybe just comment it out and see what happens maybe we're going to need it later on but I think we won't need it maybe we'll need the sequence length and the step size but I think we're not going to need all of these things here all these things here so basically we're loading them all and the next step is to now somehow get the predictions of them all and convert these predictions into text generation because what our model does is it takes a sequence and predict the next character we need to somehow make it generate text and for this we're going to need a helper function which is copied from the Kara Satori so I'm going to paste this function right now this function is copied from the official Kara's tutorial you'll find a link in a description and this function what it basically does is it takes the prediction of our model and picks one character so Ramallah what it does is when it predicts the next character we have the softmax result of the different probabilities for each character and this sample function just chooses one of them and depending on the temperature this pick this choice will be either conservative or experimental so if we have a high temperature we're going to choose a character that's a little bit more risky a little bit more experimental and if we have a very low temperature we're going to go with a safe pick so the higher the temperature the more creative the sentences but maybe they're also not going to make a lot of sense this is the trait that we have to make here and this sample function is basically the function that we just used to pick one specific character and we're going to use it in the next function that will be the text generation function so the text generation function is actually not that complicated but it's kind of long so we're going to start by defining the function generate text and this function takes two parameters length so the length of the text that we want to generate and also the temperature which we're going to directly pass to the sample function so temper temper richer yeah should temper it true sure should be fine okay this is the function generate text and in this function what we're now going to do is we're going to start with the text with the a starting text and this starting text is going to provide the the input for our neural network to predict the first next character so basically we're copying the first 40 characters directly from Shakespeare's text and everything that follows is generated by by our neural network if you want to have a text that is completely generated from the neural network or by the neural network you would have to cut off the first 40 characters because they represent the base or the basis for our text so what we're going to do is we're going to define a start index and this start index will be pretty random so we're going to say random dot random integer and this random integer shall be in between 0 and the length of the text - the sequence length of course because we need to have enough characters to you we need to have at least 40 characters so if I start at the last digit at the last character sorry at the last index I'm not going to have enough characters to to provide the input and actually minus 1 because we're dealing with indices here so this is the starting index so somewhere in the text we'll pick an entry point and take the first 40 characters so we're going to say generated text equals empty so an empty text and this text is actually what we'll end up with and we're now going to do is we're going to say sentence so the starting sentence is the text and from the text the start index up until the start index plus the sequence length so we pick this entry point and the next forty characters and this is the sentence that we start with now we get into the prediction part so first of all we add this sentence to generate a text because we need to work with the generated text over and over again since we're going to add one more character then take the generated text as a basis and so on so we're going to say sentence as I said if you want to have a completely generated text you need to cut off the first 40 characters because otherwise you have copied text from the script from the training data story so basically we do that and then we start a full loop for I in range of the length as long as or not as long as but we need to basically produce as many characters as the length parameter says so if we want to have 200 characters we're going to need to to iterate over this loop for 200 times and what we're going to do now is we're going to define the X values and the X values are actually just the and a numpy array of zeros again we're going to basically do the same thing that we did up here we're going to define a new numpy array full of zeros then we're going to take the sentence that we just picked from a random position in a text and convert this into a numpy array so we're going to say NP zeros are with the format 1 sequence length and length characters so basically the amount of characters and this is our empty or filled up with zeros numpy array and we're going to now say 40 and character in enumerate sentences we're going to say are actually numerate sentence sorry for each character and its index in the sentence we're going to say X of zero at the certain position the certain character converted into an index character will be one so basically we do the same thing that we did up here we define an empty are not empty sorry a and numpy array full of zeros and then we put a one wherever a character occurs we talked about this before so we're just going to do the same thing again and after we've done that what we're going to do is we're going to define the predictions the predictions are going to be model dot predict and the mall is going to predict on the X values and we're going to set the verbose parameter to zero and going to pick the first index of that so basically we predict the the possible characters we get the softmax results of the prediction and what we do with that is we have to feed that into the sample function so we're going to say next index equals sample prediction predictions and temperature so what we're basically doing is we're getting the predictions from our model directly which is a numerical format and then we put these predictions into the sample function with the given temperature that we have in the parameter and this gives us the next index so basically the numerical representation of the next character that we chose so now last but not least we need to convert this into the next character so we're going to say next character equals index to character or index of char next index and now we end up with a character and what we're going to do now is we're going to add this character to the generated text so that we have the next character predicted and then we're going to use that as our the next input so we're going to include this next character that we just added into the next input so that we start to generate sequences of or sentences basically so we're going to say sentence equals sentence cut off the first the first character replaced basically or basically basically left shift the whole sentence and add the new character at the end so we're basically saying the sentence from the first letter on plus the next character and in the end when all of this is done when we did this length amount of times we can return to generate a text and this is actually the text generation function so now that's basically it we can now just go ahead and print some results here so we're going to say print generate text and now we can choose a couple of different settings here so we can say for example 300 and temperature of 0.2 and we're going to try a couple of different temperatures here so let's start by saying print and here we're going to print 0.2 just so we know what temperature we're dealing with and then just copy that a couple of times and we're going to say 0.4 0.6 0.8 and maybe we're going to try a one as well so let's change these values here 0.6 0.8 and or let's say 1.0 I hope my script is not going to crash the recording again but since we're not training I think this should be fine maybe I'm wrong because it still needs to generate the text well let's see starts generating something 0.2 or actually let's just skip to the part where all the generated text is finished all right there we go we have the results so with the temperature of 0.2 we got something like Camilo I shall be the seams and the shall be is a death that the will the will and the will the world so yeah Lady Capulet or sappy I don't know how this is branched I shall not the she'll be the stay the Seif yeah makes sense let's look at 0.4 from the world stay a call the will that wouldst okay and all the see may lies a wind the brown Romeo I will shall not tell the soul this dress Strief I know I will not stay this is the deed the not come to be not to the hats alright 0.6 Juliet then then half our our light with throw ice ears John off whatever say the stand fight a drunks Polina no sir the toe with here free low and beard soul 0.8 we see something like Benvolio now counting all to is I dealing in just Camilo I have by my do s my s on my life whatever and then we got one let's start here and thickeners now be hell a Leah okay this does not make sense at all so as you can see this is a pretty high temperature already I think probably one of the best one is 0.6 0.8 maybe for lagging winters and file2 no out short I don't know even though the text doesn't make a lot of sense it is kind of impressive I think if you imagine or keep in mind that a computer doesn't know what a sentence is doesn't know a word is basically so even though we recognize the and will and ant and world as words the computer doesn't know that these are actually words it just produced them based on the training data it saw and it has also some kinds of dialogues here I don't know if they make a lot of sense but as you can see this is quite impressive for a neural network that we just trained in less than an hour I think this video shouldn't be longer than an hour and actually you can see that this is actually quite impressive so this is how you generate text with recurrent neural networks in Python now this was the tutorial on text generation which tends to flow into recurrent neural networks I hope you enjoyed it I hope you learned something if so hit the like button and leave a comment in the comment section down below now of course to this project you can also add some other text so for example you can not only scan Shakespeare text but you can actually go to whatsapp and export for example your whatsapp chats and then scan those or load those into the neural network and train your neural network to produce text that is similar to your lots of conversations if you want me to make a video on that write it in the comments down below then maybe I'm going to do it but other than that that's it for today so thanks for watching this video subscribe to this channel if you want to see more and see you in the next video [Music]