Unicorn AI - Computerphile

Key Takeaways

in the previous video we were talking about transformers this architecture that uses attention to give unprecedentedly good performance on sort of language modeling tasks and some other tasks as well but we're looking at language modelling and that was in preparation to make a video about gbg2 which is this very giant language model that has been that was recently well was recently not released actually by open ai the way that they generated the data set for this is pretty cool to get enough text they went to reddit and they pulled every website that is linked to from reddit do we have any idea of how many that is lots it wasn't literally every it was everything that had more than three karma i think or maybe more than two commas something like that like anything that had somebody had thought to poster at it and at least two or three people had thought was good enough to upvote they scrape the text from that it's pretty much just a transformer it's not the the architecture is not especially novel they haven't done any like amazing new um new discovery but what they realized was transformers it seems like the more data you give them the better they do and the bigger you make them the better they do and everything that we built up until this point is clearly not like we haven't hit the limits of what this can do um we they they thought well we think we're probably bottlenecked on data and maybe network size so what happens if we like turn that to 11. what happens if we just give this all the data and make a really big one it makes sense to talk about the acronym right so it's a generative pre-trained transformer so generative same as generative adversarial network it generates outputs it generates samples you know pre-trained is this thing i was talking about all of the different things you can use a language model for right you can do you can do translation you can try and resolve ambiguities you can do summarization you can answer questions you can um use the probabilities for augmenting other systems so yeah there's a bunch of different benchmarks for these different tasks uh that you might want your language model to do and this is what we talked about in the gridwells video of having these like standardized problems with standardized metrics and standardized data sets so that if you're comparing two different methods you know that you're actually comparing apples to apples um and this is like very important it gives you numbers on these things it's often quite difficult expect to like you're generating samples of text and it's like how plausible is this text how realistic does it look like how do you put a number on that it's kind of difficult so there's all of these standardized metrics and the thing that uh people came to realize which actually i mean i say that is though it's like some amazing discovery it's fairly obvious if you train your system in like an unsupervised way on a large corpus of just general english text and then you take that and train that with the data from this benchmark or the data from that benchmark you can like fine tune it so you start with something which has like a decent understanding of how english works more or less and then you say now i'm going to give you these samples for like question answering or i'm going to build a system using that to solve to go for this benchmark so it's pre-trained you start with something that's like a general purpose language model and then you from that a fine-tune it to whichever actual benchmark or problem you're trying to solve and this can give you better performance than just starting from nothing and training to each of the benchmarks from scratch makes sense and so the point of the gpt2 paper the thing that makes it cool is they said okay if we make a really huge one what if we don't fine-tune it at all what if we just make a giant model and then just try and run it on the benchmarks without messing with it without showing it any of the specialized data for that benchmark just the raw general purpose language model how does that perform and it turns out surprisingly well so this is a very very large data set for text it's about 40 gigabytes which actually doesn't sound like very much but like for text text that's insane right it's uh somebody said that this was the size of google's entire index of the internet in 98. so like it's yeah it's a lot of text um and they trained it on that and they ended up with a 1.5 billion parameter model but which is like a previous state-of-the-art system with 345 million this is 1.5 billion so they've just made the thing much much bigger and it performs really well some of the samples that they published quite uh captured the public imagination you could say and now that we've talked a little about the problems that neural networks or any language model really has with long-term dependencies we can now realize just how impressive these samples are because when you look at them as a you know if you look at them uninitiated you're like yeah that's pretty realistic it seems to like make sense and it's cool but when you look at it knowing how language models work it's like very impressive the the coherence and the consistency and the long-range dependencies so we can look at this one that got everybody's attention the unicorns one right so they prompted it with in a shocking finding scientists discovered a herd of unicorns living in a remote previously unexplored valley in the andes mountains even more surprising to the researchers was the fact that the unicorns spoke perfect english well hello and from there you then say you go to your language model gpt2 and you say given that we started with this what's the next word and what's the word after that and so on so it goes on the scientists named the population after their distinctive horn of its unicorn these four-horned silver-white unicorns were previously unknown to science we do have a clue here as a human being unicorn forehorned doesn't quite make sense but nonetheless we're going okay now after almost two centuries the mystery of what sparked this odd phenomenon is finally solved doctor would that be jorge jorge perez j-o-r-g an evolutionary biologist from the university of la paz this is impressive because we've mentioned the andes mountains in our prompt and so now it's saying okay this is clearly you know in a shocking finding this is a science press release news article it's seen enough of those because it has every single one that was ever linked to from reddit right so it knows how these go it knows okay third paragraph this is when we talk about the scientist we interview the scientist right okay first word of the scientist paragraph doctor obviously right because this is the now we're in the name of the scientist what name are we going to give it needs to be a name conditioning on the fact that we have the andes mountains so we need to get we're in south america uh the name probably should be spanish or maybe portuguese so we get we get dr perez here and then evolutionary biologist makes sense because we're talking about animals from the university of la paz again this is the first sentence like when you have that first clause that introduces the scientist you always say where they're from so we say from the university of and then university names tend to be the name of a city what's a city where we have the andes mountains so we're gonna you know bolivia la paz perfect and the thing that's cool about this is it's remembered all of these things that were quite a long time ago several sentences ago well it hasn't remembered them it's paid attention to them across that distance which is impressive but also this is encoding a bunch of understand understanding a bunch of information about the real world right all it was given all it knows is statistical relationships between words but the way that it comes out to us is that it knows where the andes mountains are what kind of names people in that area have what their cities are what the universities are all of those facts about the real world because in order to have a really good language model it turns out you have to kind of implicitly encode uh information about the world because we use language to talk about the world and knowing what's likely to come next requires actual world understanding and that's something that we see in some of the other things that they got it to do you can see the real world understanding coming through let's keep going university of la paz and several companions were exploring the andes mountains when they found a small valley with no other animals or humans perez see we're hanging on to him yep we're referring to him again uh but now we've changed it to be just the surname because that's the format that people use in news articles perez noticed that the valley had what appeared to be a natural fountain surrounded by two peaks of rock and silver snow presently others then ventured further into the valley around about here in our article we should have a quote from the scientist right quote by the time we reached the top of one peak the water looked blue with some crystals on top and we're talking about this fountain i guess this natural fountain we're referring back to the previous sentence like everything is relying on and contingent on earlier parts of the text while examining i've skipped a paragraph while examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular english now when i read that i was like okay this is now unusually good because that's the second sentence of the lead right we're six paragraphs in and it knows about this point i've covered the first sentence of this initial paragraph now it's time to talk about this second sentence of the lead even more surprising to the researchers of the fact that they spoke english and it completely ignored the speaking english part until it got to the part of the news article where that comes in you've gone six whole paragraphs the idea of accurately remembering that the unicorns speak perfect english is like that's very impressive to me and then it goes into it's gets a little bit unhinged starts talking about it's likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through dna that's reddit right well it's not actually stuff on reddit it's stuff linked to from reddit but yeah this is this is news articles man they seem to be able to communicate in english quite well which i believe is a sign of evolution or at least a change in social organization said the scientist that's his evolutionary biology though right right right yeah we know he's an evolutionary biologist so so the the coherence of this text is really dependent on its ability to condition what it's generating on things that it's generated a long time ago so yeah so it can generate really nice news articles and it can generate all kinds of text things that it anything that is sufficiently well represented in the original data set so that's gpt2 it's a really unusually powerful and like versatile language model that can do all of these different natural language processing tasks without actually being trained specifically on those tasks um to really and that's that's why it's impressive it's not that it's a it's a brand new architecture or a brand new approach or whatever it's just when you make these things really huge and give them tremendously large amounts of data uh the results are really impressive in the original data set so it will it will write you lord of the rings fan fiction it will write you cake recipes like there's all kinds of examples of different samples here's a recipe for some kind of peppermint chocolate cake and it's got a bunch of different