The Science Behind Google Translate: Understanding Transformers

Key Takeaways

have you ever asked yourself how does Google Translate work well the answer might surprise you but it doesn't use any CNN's they are done using transformers and I'm not talking about those in the movies let's dive into this translation mystery recurrent neural networks also known as RNs are widely used in natural language processing but they are slow and can deal with long sentence very well since they work only one word at a time it leads to problems like vanishing or exploding gradients if you are not familiar with rnns and NLP I explained them in two videos which are linked in the description I suggest that you take a look before continuing this video transformer neural network architectures were introduced in 2017 with the paper attention is all you need by Google to get rid of those RN ins and CNN's models and just use attention instead using a similar approach to errand ins transformers will make the use of more paralyzation possible during training making it way faster and more accurate first we need to look at attention since it's a key ID and transformers attention is the evaluation of the relationship between each input item to each output item then there is self attention which is the evaluation of the relationship between each input item to every other input item this is the attention with respect to oneself it tells us how a particular word in the sentence relevant to other words in the same sentence transformers networks use self attention multiple time called multi-headed attention each headlines attention relationships and dependently transformers are a sequence to sequence model that uses encoders and decoders and they work much like our enhance it uses multiple multi headed attention modules stacked on top of each other the encoder evaluates the degree of relevance of each word with respect to the other words in the sentence to be translated the encoder encapsulated both language and directions like French and English interactions doing that we get the relationships with other words in both languages overall the decoder predicts the next word and we execute this over multiple time steps until the end of the sequence these encoders and decoders are all connected to feed-forward layers transforming the output to make it digestible by the next encoder or decoder block the main difference with our hands is the fact that input sequences are passed simultaneously instead of one at a time which is much faster for example in natural language processing are an ends line one word at a time while transformers can learn all words of a sentence simultaneously it is used in sequence to sequence tests like machine translation where it takes a sequence of words for example in English and translates it in a sequence of words in another language like French like everything transformers have benefits and disadvantage they are faster training time than our enhance if you have access to sufficient compute because of the paralyzation plus the attention mechanism ignores order which means that it is as easy to detect relationships between very distant item as it is to the tag relationships between close item in a sequence however there are drawbacks as well transformers are very large models so it needs a lot of memory and compute to Train and requires a lot of data finally since they are pretty new we know less about them and what works like the effects of the hyper parameters compared to the RN ends and cnn's this was just an introduction feel free to check the links in the description to learn more about transformers please leave a like if you learn something and subscribe to the channel to not miss any further term clearly explained [Music]