What is Recurrent Neural Network (RNN)? Deep Learning Tutorial 33 (Tensorflow, Keras & Python)

Key Takeaways

so far in our deep learning tutorial series we looked at artificial neural network and convolutional neural network which is mainly used for image processing in this video we will talk about recurrent neural network which is used mainly for natural language processing tasks so if you think about deep learning overall cnns are mainly for images rnns are mainly for nlp there are other use cases as well so we'll understand how recurrent neural network works and we'll look at different applications of rnn in the field of nlp as well as some other domains we will be looking at some real-life use cases where sequence models are useful you must have used google mail gmail here when you type in a sentence it will auto complete it so just see when i type not interested at this time is something it auto completed so google has this rnn or recurrent neural network embedded into it where when you type in a stance and is not interested at it will auto complete with this time if you say we'll let you know if it changes it will also say in the future so this saves you time it will write the sentence for you another use case is translation you must have used google translate where you can translate sentence from one to another language easily third use case is named entity recognization where in the x you know you give neural network a statement and in the y neural network will tell you the person name the company and time rudolph smith must be a millionaire with tesla's prizes skyrocketing so these are various use cases where uh using sequence models or rnn recurrent neural network helps the fourth use case is sentiment analysis where you have a paragraph and it will tell you the sentiment whether this product review is one star two star and so on now you would think why can't we use a simple neural network to solve this problem see all these problems they are called sequence modeling problem because the sequence is important when it comes to human language sequence is very important for example when you say how are you versus you are how doesn't make sense right so the sequence is important here and you would think why don't we use simple neural network for that well let's try it so for language translation how about we build this kind of neural network we know where input is the english statement and the output could be in this statement once i build this network what if my sentence size changes so i might be inputting different center size and with a fixed neural network architecture it's not going to work because you have to decide how many neurons are there in the input and output layer so with language translation number of neurons becomes a problem like which what do you decide as a size of neurons now one would argue okay i would decide let's say a huge size let's say 100 100 neurons and remaining if i'm saying did you eat biryani so it will occupy 4 neuron remaining 96 i will just say 0 or you know blank statement that might work but still it's not ideal the second issue is too much computation when you all know neural networks work on numbers they don't work on string so you have to convert your word into a vector so one of the ways of converting that into a vector is let's say there are 25 000 words in your vocabulary and you will do one hot encoding where you know how let's say is at 46 position r is let's say second position u is let's say at 17 000 position so at that position you put 1 remaining position you put 0 and that's called one hot encoding you have to do similar thing for output as well but you realize this will increase too much computation each of the word when you convert it to a vector you know how many neurons you need in the input layer itself is humongous the third issue is this sometimes when you translate language you for let's say two different english statements you might have a uh hindi statement so in this case uh when i say on sunday i ate gold kappa let's say i train this network based on this statement and then for on sunday let's say it will adjust the weights of all these edges which i have highlighted in yellow color same statement i can say differently i can say i at golgappa on sunday so now on sunday the meaning of on onsen is same but here neural network has to learn different set of edges you see all these edges are in yellow color so the parameters are not shared we looked at in our convolutional neural network tutorial as well that by using convolution operation we can share the parameters here the use of a n or artificial neural network doesn't allow you to do that okay also the most important part in all this discussion is the sequence see when you have structured data for example you're trying to figure out if the transaction is fraud or not and let's say your features are transaction amount whether the transaction was made out of country or whether the assassin that customer provided is correct or not now here if you change the order of this features let's say ssn correct i supply you know my first neuron it's not going to affect anything you know because the sequence in which you supply the input doesn't matter whereas if you have english trans english to hindi translator translation and instead of saying i ate gold ponce and if i say i at sunday on golgotha the meaning becomes totally different so now you cannot say that the hindi translation is high because it becomes invalid so sequence is very very important that's why artificial neural network doesn't work in this case just to summarize these are the three major problems with using a n for sequence problems let's once again talk about name entity recognition let's say the world loves baby yoda i love my baby grogu i love mandalorian series and we have got this nice uh baby grogu at our home which actually talks with us in this statement devil and baby yoda are person names okay so the whole purpose of name entity recognization is to find out the entity you know like double as an entity is a person baby yoda as an entity as a person so that's the whole goal of ner now you can represent this as ones and zero so if the word is person name you would mark it as one and it you can take a vocabulary and use one hotend coding and there are other ways of vectorizing a word then you have a layer of neurons so these are all individual neurons let's say this is one layer it's a hidden layer you supply that and you get one output okay so each neuron all you know has a sigma function and activation function so now while processing the statement the world lost baby yoda now i will process it word by word so i supply the world get the output and then i go back again now i supply laws converted into vector and the previous output which i got which was y double ins of supply that as an input to this uh layer so you see the input of the layer is not only the next word but the previous output because the language makes sense language needs to carry the context if i have just a word loss and if i don't have double in front of it it it might mean a different thing so there is a context that you need and this kind of architecture provides your context or a memory in the third world again you supply baby to the same network right so our network has only one layer it has only one layer so there is input layer output layer and the hidden layer is just one and it has bunch of neurons in that we are repeatedly processing word one by one okay and you keep on doing this now the benefit of this is when i'm processing baby when i get why loss that while house carries the state the previous state or previous memory of double loves you know the whole statement now i'm presenting this in a different way make sure these are not four different hidden layers this is not this is a time time travel okay so actual hidden layer is only one i am just doing a time travel so first when i supplied world the wall i got this output and output was nothing but the activation function which i am denoting with a1 and you need some previous activation a0 as well let's say it's a vector of all zeros then you supply second word loss and use the previous output which was y double so y double and a one they are both same here and then you get another output a two where that you supply along with the third word baby to the same network so this these four neurons it's the same single layer i am just showing different the status of it at different times okay so you have to be very clear on this that these are not four different layers it's just one layer just because i am showing different time steps that's why i'm i'm showing you almost a time travel here and once the network is trained of course it will output like double is one loss is zero baby is one and so on okay so you get your ner output uh individually here one other way of representing the same network okay because just to avoid confusion and to make presentation little more clear many times in literature you will see presentation like this where each word which is an input comes from the bottom and there is activation so again this and these two diagrams are exactly same okay i'm just putting this word at the bottom generic representation of rnn is this so this is the real representation you have only one layer and you are you are kind of almost in a loop you are you are supplying the output of previous word as an input to the second word so now let's talk about training so again the problem we are talking about is ner where these are my training samples okay x and y x is a statement y is whether a given word is person name or not so we are processing first training sample develops baby yoda so this one i will first initialize my neural network weights with some random values then i supply each word then i calculate y hat which is predicted y then i compare with the real y so real y here is 1 0 1 1 so i compare that with here so 1 0 1 1 i compare that with y hat and i find out the loss okay and and then i i sum the loss so that will be my total loss you all know about grade and descent right so we compute the loss then we back propagate the loss and we adjust the weights so now i take the second statement iron man punched on hulk's face he was very angry with hulk again i calculate all the losses then i find total loss and then i do grid and decent to reduce the loss so i keep on doing this for all my training sample less i have 100 training samples passing all hundred training samples through this network will be one epoch we might do let's say 20 epochs and at the end of the 20 epoch my loss might become very minimum at that point we can say my neural network is trained let's take a look at language translation so in language translation what happens is you supply first word to your network then you get the output then again same network you supply second word and the output from previous step as an input and of course when you supply first where you have to pass in some activation values let's say all a vector of all zeros then you supply third word for fourth word and so on and when you're done with all the words that's when the network starts to uh translate it because on you cannot translate one word by one because after the statement i can push maybe one more word and that will just totally change my translation that's why for language translation you have to supply all the words and only then the network can translate for you so the network will translate it like this and the first part is called encoder the second part is called decoder we will go more in depth into all this but i want to quickly demonstrate how the neural network looks in the case of language translation now this layer doesn't have to be just single layer it can be a deep rnn as well where the actual network might have multiple hidden layers okay so i hope that clarifies uh the architecture behind rnn and you understand why you can't use simple neural network here and you have to use specialized neural network called rnn which can memorize for you which can remember previous state because language is all about sequence if you change the sequence the meaning changes so uh if you like this video please give it a thumbs up and we'll be having more recurrent neural network and nlp type type tutorials in the future videos thank you