75 years of Language Models

Key Takeaways

in this video we're going to mention 20 of the most important research papers that made language models as powerful as they are today we're going to distill this by covering research since the 1940s so let's get to it first off language models can predict the next word in a given sentence given its previous words and these models don't actually process words but they process numbers and so we need a way to represent language with numbers so that these models can understand them and eventually they'll be able to perform intelligent tasks and this is the basis of language processing with that the 20 papers that we're going to go through will represent language with numbers in the most effective way as possible so let's get to it we start off with the paper mathematical theory of communication this paper was written in 1948 by Claude Shannon who is regarded as the father of information Theory so essentially with this paper since we want to represent language with numbers we do so by representing sentence with n gram vectors this is done by first breaking down the sentence into its corresponding engrams and determining which engrams are present in the representation this was one of the earliest ways that computers understood sentences the second paper is the one titled neural probabilistic language models so an issue with the engram vectors approach is that they were sparse vectors so they're too large for computers to process this paper introduced the idea of neural networks to solve the problem of language modeling and during this training the model learns also how to represent words with a dense fixed size Vector which may be of size like 64 128 or 256 Dimensions this is a lot smaller in Vector size than like the n-gram vectors the next paper here is NLP almost from scratch so until now we would need to learn vectors by training neural networks on language modeling or some other task but if we change the problem then we would need to train these word vectors from scratch so instead of training a neural network on a specific problem why don't we train it on multiple problems with a unifying architecture and so different problems could share the same word Vector representation and this is somewhat a philosophy that we even follow today now in this research the core architecture uses convolution to learn vectors this was inspired by previous research on time delay neural networks introduced in 1989 for phoneme detection so overall the NLP almost from scratch paper introduces the idea of learning shared representations of words next up we have the efficient estimation of word representations in Vector space now this is the paper that introduced word to VEC so keep in mind that the goal even since the 1940s paper was to represent text that is like words and sentences with some Vector that preserves meaning and is tractable for computers to understand the previous architecture is an NLP almost from scratch does learn word representations that are reasonably high quality but it needs a lot of data and a complex architecture to do so word tovec learns word vectors but with a much simpler architecture word to VEC itself is a framework of two architectures one is C bowel and the other is Skip Graham we train this network by passing in and also passing out one hot Vector of words but this Center layer is going to have the internal dense word representation that will eventually be learned and so once it's trained we're going to have a dictionary of every word followed by its Vector representation now a similar paper is glove which is global vectors for word representation this is similar to word to VEC but the vectors are formed instead of just using local context information we use Global co-occurrence of words in order to form the final dense word representations another paper that is very similar also is enriching word vectors with sub word information this is the paper that introduced fast text which is also similar to word to VEC but it's very useful in morphologically Rich languages like Finnish Arabic and some Indian languages morphologically Rich languages have words that can change their forms slightly just based on gender or preposition if you want more information on word to VEC love and fast text I have a full video as well next is a convolution neural network for modeling sentences so we've spoken about words and representing words as vectors but we also want to represent present sometimes sentences as a fixed length vectors such that their meaning is preserved so this dcnn does so using a convolution and some Max K pooling operations and this K really depends on the size of the input and hence this is a dynamic Network and this way longer sequences of words can be represented pretty well for more information on Dynamic convolution neural networks and also time delay neural networks you can check out my companion video here now the next paper I want to introduce is learning internal representations by error propagation so this is the paper that was written in 1985 and introduced recurrent neural networks so recurrent neural networks like Dynamic convolution neural networks can process sentences they get representations of sentences and can even solve tasks involving these sentences this dissertation is laced with however a ton of math and so I would recommend actually reading chapter 10 of the deep learning book because it actually gives a good intuition about recurrent neural networks from scratch and also has very nice visuals to accompany it next up is long short term memory the main issue with traditional recurrent neural networks is that the gradients would either explode or vanish so the lstm architecture was introduced to combat this issue for longer inputs now the outline of this paper is kind of clear but there is a lot of math laced into it as well and so for this case I would also recommend a popular blog post on this topic by Cola and I think they explained lstm's very good with some stunning visuals for additional information I have a full companion video on this topic too and yes I make a lot of videos now I also wanted to call to attention a PhD thesis by Ilya skever the premise here is the original rnns and lstms that were introduced in the 80s and 90s didn't quite gain popularity until past the 2010s a big reason for this was that they were very tricky to train and this thesis showed how they can be stably trained using some techniques on parameter initialization optimization and regularization and this is why recurrent neural networks for NLP became super popular around like the 2013 to 2016 time next up we have deep contextualized word representations so let's pivot back to word representations like using word to VEC now the goal is to understand really high quality embeddings for words now the main issue with word to VEC is that it really doesn't care about contacts the same Vector is generated no matter where the word is in the sentence but this isn't good because the word meanings can change but the vector doesn't to solve this Elmo considers contacts so every word Vector that is generated is a function of all words in the sentence and it does so by having a bi-directional STM architecture so lstm networks help them train on Long sequences in the bi-directional component allows Elmo to understand words that come before it as well as words that come after it the result is a deep contextualized word representation hence the paper and we can fine tune this on any other NLP problem now the next paper I want to talk about is a very famous one called attention is all you need another issue with lstms and rnns are that they are slow to train and this is because data needs to be passed into them sequentially the architecture consists of an encoder and a decoder that makes use of attention for processing long-term dependencies kind of like lstms now this transform architecture was kind of the turning point from the recurrent base architectures to more attention-based methods that set the stage for a modern language models for a brief overview on this topic I have a video and also to code this out completely from scratch I have a playlist of videos more fun more homework next up we have Bert the pre-training of deep bi-directional Transformers for language understanding so Bert essentially is a stack of Transformer encoders the main issue with Transformers is that we need to train the architecture from scratch every single time that we need to train on a new task which is also language related so language modeling question answering machine translation each of them we need a lot of examples of each and this may not be super easy to get your hands on so to deal with this Bert split it's training phase into two parts one is a pre-training phase where we train on language modeling and next sentence prediction and then we fine tune it on whatever NLP task that we want to do and the idea is that with fine tuning you don't need as much data next up we have improving language understanding by generative pre-training this is the paper that introduced GPT so GPT is a stack of Transformer decoders that also leverage transfer learning in order to learn language tasks it dives into the same pre-training and fine-tuning phase that we also saw with Bert next up is the paper language models are unsupervised multi-taskers this is the paper that introduced gpt2 so the older versions of gbt and Bert still required quite a bit of data for fine tuning overfitting is also easy and honestly the fine tuning process is not exactly how humans would learn we don't need like hundreds of thousands of fine-tuned examples just to understand how to translate from one language to another or how to complete the sentence we might need just like one or a few examples and this is where meta learning such as like zero shot learning One-Shot learning and few shot learning come into play now while gpg2 wasn't super successful this paper called the language models are few shot Learners that that introduce gpt3 was pretty successful so gpt3 is essentially a scaled up version of gpt2 which has a suite of architecture such that its largest architecture is 175 billion parameters now for more information on gbt one two and three and its progression I suggest you check out this video next up is sentence Bert sentence embeddings using Siamese Bert networks this is the paper that delves into a new kind of architecture called sentence Transformers so Burton gbt produce high quality word embeddings but how do you get sentence embeddings from this if you simply take like the average of the word embeddings that you get and just use that average Vector as a sentence embedding well that's not very high quality instead we would want to train a Siamese network of birds in a pooling layer on a few tasks so one of them is natural language inference where we check if one sentence either entails contradicts or is neutral with respect to another sentence and the other task is sentence text similarity where we have two sentences and want to see how similar they are to each other once these networks are trained we can then just use that Bert plus pooling layer in order to input the words of a sentence the pulling layer will take some form of weighted averaging and then it's that Vector that is going to be much higher quality in representing a sentence the next recommendation here is the chat GPT blog chat TPT is probably one of the most well-known items on this list and it is a fine tune chat bot that makes use of reinforcement learning with human feedback the original GPT architecture is fine-tuned to answer questions we then build a rewards model that says given a question and an answer give a score that says how acceptable this is within use reinforcement learning to fine-tune the GPT model and this is done to ensure that chat TPT is safe factual and non-toxic and I have a playlist of videos explaining this infographic too and next up here we have llama so this is the main paper for llama two llama two is to gpt3 as llama to chat is to chat gbt so llama itself is a pre-trained language model it is open source architecture that is trained on publicly available data compared to GPT it has a smaller architecture but is trained on much more data and because of its small architecture inference becomes faster and for more information on llama I have another full companion video that's all for today I hope these papers helped you navigate the idea of what papers are important in the fields of language modeling and natural language processing in general if you like the video please do give it a like subscribe and I will see you very soon in another one bye