Why BERT and GPT over Transformers

Key Takeaways

Transformers versus Bert and GPT Transformers have a few disadvantages they weren't designed to be necessarily language models they were designed to be sequenced two sequence models sentences just happen to be sequences of words so we can technically solve language problems with Transformers but even so language is complex Transformers would need a lot of data to learn these representations from scratch and they might need to become more complex in order to capture the essence of language this is where Bert and GPT shine Bert is a stack of Transformer encoders GPT is a stack of Transformer decoders they were both designed to understand language in their pre-training phase and then fine-tune's specific tasks during their fine-tuning phase