The Illustrated Retrieval Transformer

Key Takeaways

language models are able to generate text better than any software system we've had before and the larger the model the more the training data it was trained on the better the text that it generates there's a challenge here however in that these large models need super computers being able to run them and deploy them vast amount of resources go into that a lot of know-how is required to get that working so the question is how can we make these models smaller one of the leading ideas to empower smaller models to have great text generation capabilities is retrieval this is the idea that the model does not need to store the world's information into its parameters and when deployed we don't need to load all of that information into GPU memory retrieval language models boost the model with a database or the ability to retrieve information from a search engine like Google and the retrieved information from that database or from the internet helps it generate text in this video we'll be going over my article explaining deepminds retro Transformer which at four percent the size is comparable to gpt3 at language intensive tasks this is a very exciting direction that is explored in retro Transformers but also in other models like memorizing Transformers like web GPT and even Google's Lambda honestly this is an idea so powerful that it almost seems wasteful for future language models to be created without this this ability it also has nice properties like the ability to curate and filter the data that the model has access to and potentially also the ability to update the model's data without the need to retrain the model from scratch with that let's look at retrieval Transformers so these cool developments in retrieval and the ability to empower Transformer models with a retrieval or a search component is very exciting and it's we saw a couple of these papers towards the end of 2021 this being one of them the Retro Transformer being one of them web GPT being another one and then memorizing Transformers was released after that and they were all around this very cool idea that you do not need a model with 175 billion parameters it's not 85. it's you don't need to store all of the world's information in the parameters of the model and reload them into GPU memory you can have a Model A language model four percent of the size but then help it with a retrieval component and that injects information from other sources from the Internet or from a database in this case into this model and so um here's an example of let's say we have this prompt that we give the model and the model needs to complete it so if we say the Dune film was released in this is something that the model has to have this factual information just knowing knowledge is not enough for this but here's another example of a of a prompt it's popularity spread by Word of Mouth to allow Herbert to start working full you don't need factual information to fill this in you need maybe just you have to understand the probabilities of words and from here I there are these two components potentially of of language models that are both represented in these large GPT models so language information and then World Knowledge information and I love that this new approach of retrieval your language model needs to maybe have a focus a little bit on language information and the world knowledge information can be outsourced and it's better for all of that information to not be stored within the model so you don't need to train the model every time you upload it so we can use Advanced and high performance databases for example to retrieve very quickly instead of inefficiently storing all of that into this massive model so this is I think a visual that captures a little bit of why this is exciting and why after this comes out it seems that this is really the way to go um and it sort of helps you audit the sources of this information and you can have a little bit more more control over where this information was sorted from and what information sources does a model have access to I'm really excited about this area here we talk a little bit about how this one model does it through his Retro by by Deep Mind so it has a built-in database of keys and values and then when it gets an input prompt it gets the tokens and makes it creates a sentence embedding of that and then it uses that as a query to see where has the models database seen words or sentences that are similar to this so this is the retrieval part it uses the prompt the input prompt as a as a query into the database and then from there it gets two things it gets this similar sentence that is similar to the input but also the sentence after it so to speak because that's maybe a lot of the times where that information would be would would and so these are all fed into a retro so it gets the input prompt but it also gets this relevant information and that this system has deemed let's say relevant to this so it provides a number of different neighbors I think it's to a training time but then at you can do more like you're not you're not limited to that a little bit more uh details about the actual architecture or structure so you have a encoder stack and a decoder stack and the encoder is mainly to encode the the neighbors into let's say keys and values and then when you have the prompt that goes into the decoder and specific layers in the decoder specific decoder blocks this retro decoder block has access to this encoder through the keys and values so these layers layer number let's say nine here can look back at the relevant information and brings the relevant information from the database memorizing Transformer does it a little bit you know in a different way maybe a way that's maybe simpler than the one specifically used here but this is a very interesting direction that I super excited to see us taking because it allows us to use much smaller language models and have other better retrieval tools to augment its information