Introduction to Transformers

Key Takeaways

transformers have been taking the nlp area by storm libraries like hugging phase has made it very easy for everyone to use transformers or implementations like bert or gpt3 is the reason that everyone is talking about so in this video we will look closely into transformers and understand their working principle for transformers word is common we were using rnns to deal with text daily but the problem with rnns is that when you give it a very long sentence it tends to forget the beginning of the sentence when it comes to the end of the sentence and because they rely on recurrence they cannot be parallelized then we start using lstms lstms are a little bit more sophisticated they tend to remember information for a little bit longer of a time but they take very long to train well then we have transformers transformers only rely on attention mechanisms to remember things they do not have any recurrence at all and thanks to this they are faster we train them in a parallel way generally attention is the ability of a model to pay attention to the important part of ascent