GPT-3 explained with examples. Possibilities, and implications.

Key Takeaways

[Laughter] [Music] hello today we are going to discuss a very trendy topic it has made the news and it fills up my twitter feed these days and the topic is open ai's gpt gpt3 in this video we will discuss three things what is gpt3 what can gpt3 do we will have some cool apps and examples and how much do we know or not know about gpd3 and what potential problems and limitations might be let's start with the first question what is gpt3 no gpt 3 is not a time zone gpt stands for generative pre-trained transformer and is more or less a very cool and sophisticated language or sequence continuator the word transformer is referring to the transformer of 2017 invented by google 2018's gpt uses the transformer modules and has a total of 110 million parameters but 2019's gpt2 brought 1.5 billion parameters to the game the gpt language model trained on next word prediction can generate continuations of text so well that openai was afraid to release the model worried that it might be used for generating deep fakes and false texts to news well this is 2020 and this year has brought us many things but it has also given us gpt 3 that has no less than 175 billion parameters what will 2021 bring us the gpt-3 architecture is the same as the predecessors gpt2 and gpt only bigger does size matter though well kind of with many parameters the model can solve more complicated problems but with many parameters one should also train on larger and larger amounts of training data to avoid overfitting do you remember science fiction movies where the ai can finally connect to the internet and becomes extremely powerful reaching the singularity and surpassing the human intelligence well we are not in that situation yet but we have to acknowledge the sheer amount of data the 175 billion parameters were trained on gpt3 has more or less seen the whole internet and this has implications we do not grasp yet so what can gpt 3 do after it has seen so much this leads us to the next part of the video as the gbt3 paper already said it's in the title gpt 3 is very good at few shot learning few shot means that only very few training examples are needed to get good performance on the task when we were younger who are we kidding this was a few weeks ago so again when we were only a little bit younger we had to fine-tune pre-trained models like bird or gpd2 on not so small data sets with task specific annotation nowadays similar results to fine tuning can be achieved only by giving to gpt 3 some few tasks specific examples gpt 3 seems to do it all or can it though we will discuss this towards the end of the video for now let's see where gpt 3 shines while the paper discussed benchmarks or not so benchmark cases we will here see what the community on twitter could do with the api of gpd3 if you are interested to place it yourself you can request access and hope you get it few shot is the first topic here in our showcase where michael posts about his successful attempt to convert legal text into commoners english by showing only two examples to gpt3 the model was able to create even more similar results see for yourself well you could say that all this is a very good pattern matching algorithm which it more or less is so here is an example of another kind of pattern matching where paul katzen created one spreadsheet function with gpt3 which can look up state population twitter usernames and employers and do math needless to say that ai before gpt 3 could do these things but only after being trained on lots of such data and could hardly do all these things enumerated here at once not convinced yet we have more yash build a bot with gpt 3 that generates financial statements just amazing unsurprisingly since it is a language model gpt 3 is perfect for text completion like for generating automated answers to emails so you do not have to write those yourself anymore do you emails are perhaps not surprising what about poetry preserving verse and ensuring rhymes i am not so happy about this example in particular since i hoped coding was one of the last things ai would do but here it is check out these examples by sharif shameem where gpt 3 is generating code from just descriptions of what one wants to do what about chatbots with impersonation skills if you always wanted to chat with elon musk but he was too busy to chat with you you can chat with gpt3 impersonating him instead many deep fake storms ahead this beautiful example used google to extract the text from image and then process the text with gpt3 to extract ingredients find an emoji determined if it's unhealthy and give a definition o m g g p t 3 can also be a search engine if you want are there any limits one more example this time a funny one where merc's mensch cosmopol asked gpt3 about god he certainly has no questions anymore now let us speak about the elephant in the room coming to our last question what if this view shot behavior is just happening because gpt 3 has seen it all during training and now our tasks we come up with are not so new to the model at the end of the day well we just do not know and emily bender just said it i don't find anything linguistically interesting about massive language models especially without detailed information about their training data she has a point here especially we need to know what all of the training data was when thinking about harmful biases like in this example with really unpleasant results the huge amount of data means also a lot of compute power used during training the gpt 3 paper says practical large-scale pre-training requires large amounts of computation which is energy intensive training the gpt-3 consumed several thousands days of beta flops per second of compute during pre-training compared to tens of days of petaflops per second for a smaller gpt-2 model this is a problem for normal researchers especially for academia in many countries where compute resources are rather scarce the second problem is of course the environmental impact of the energy consumption during long training sessions the open ai paper brings a defense point about energy consumption the paper says though models like gpt 3 consume significant resources during training they can be surprisingly efficient once trained even with the full gpt 3 generating 100 pages of content from a pre-trained model can cost on the order of 0.4 kilowatt hour or only a few cents in energy costs this is a fair point but we ask ourselves what if this training of bigger and bigger models on even more training data becomes the trend of the future in training and developing huge models consuming lots of energy just goes on and on gpt 3 performs so well as seen in the twitter examples or in the gpt3 paper on sentence generation machine translation and so on so it's normal that the model seems very very attractive to large-scale industry and seems like the future will have a lot of applications with gpt 3 built in them well i do not want to make anyone worry about this but the gpt3 paper also shows that the model is very susceptible to adversarial attacks these attacks show how problematic it might be if gpt-3 deployed somewhere is targeted with malicious adversarial attacks more about adversarial attacks in the next video well a lot is still to be investigated here and i hope that openai will tell us more about the data it has trained the model on for a start if openai becomes more open and grants wider access to the model the ai research landscape will certainly change just an example during the last years the frequent and probably the most annoying paper reviewer question was why didn't you use bird miss coffee bean already prepares herself for the next reviewer question at conferences why didn't you use gpt 3 anyway until we know more miss coffee bean seems to agree with this opinion here which in our translation sounds like this gpt is like a small child that looked at the whole internet does not understand anything but can repeat everything and yearn help good also delivers an example where if asked in a certain manner gpt 3 responds that the sun has an eye a blade of grass has an eye too so we have seen with this example that this child does not have a lot of common sense only that this child can solve what a.i has been trying to solve for decades and this child will become some kind of an oracle i'm afraid well these are certainly interesting times to be alive you