Proof that AI Understands? 👀 Andrew Ng on LLMs building mental models, Othello GPT, Geoffrey Hinton

Key Takeaways

yeah I think it's fairly urgent for the researchers to come to a consensus about whether these big chatbots like gpg4 or Bard actually understand what they're saying um there's clearly some people believe they do and some people believe they're just stochastic parrots I believe the large language models and other large AI models are building a world model or building something that looks a lot like a world model so my guts is that I do believe to the extent this building World model is conveying learning some understanding of the world so there's been a lot of discussion recently whether or not llm models these AI models behind tragic BT and others do they understand can they reason do they think what are words you want to use the question is is our children is this an intelligence that is in some way similar to humans or is this just a very Advanced autocomplete model that's spitting out statistically likely words one after the other and he posts a suite saying do large language models really understand the world or just give the appearance of understanding and then he talks about some evidence from Othello GPT shows that llms build models of how the world Works which makes me comfortable saying that they do understand as yesterday's positive report card shows children's do learn do larger language models understand the world as a scientist and engineer I've avoided asking whether an AI system understands anything there's no widely agreed upon scientific tests for whether system really understands as opposed to appearing to understand just as such no tests exist for Consciousness or sentience as I discussed in an earlier letter this makes the question of understanding a matter of philosophy rather than science but with this caveat I believe that LMS build sufficiently complex models of the world that I feel comfortable saying that to some extent they do understand the world so Othello a board game in which two players take turns placing game pieces on an 8x8 grid I'm not familiar with this game but they do have some diagrams on the paper that they wrote so that might be helpful so during training the network saw only sequences of moves it wasn't explicitly told that these were moves on a square 8x8 board or the rules of the game after training on a large data set of such moves it did a decent job of predicting what the next move should be the key question is did the network make these predictions by building a world model that is did it discover that there was an 8x8 board and a specific set of rules for placing pieces on it that underpin these moves the authors demonstrated convincingly that the answer is yes specifically given a sequence of moves the Network's hidden unit activations appear to capture a representation of the current board position as well as available legal moves this shows that rather than being a stochastic parrot that tried only to mimic the statistics of straining data the network did indeed build a world model while this study used Othello I have little doubt that LMS trained on human text also build World models you know with that let's take a look at this paper it's called emergent World representations exploring a sequence model trained on a synthetic task and so here we have people from Harvard MIT Northeastern University language models show a surprising range of capabilities but the source of their apparent competence isn't clear do these networks just memorize a collection of service statistics or do they rely on internal representation of the process that generates the sequences they see and so they have a variant of the GPT model made to predict legal moves in a simple board game a fellow after you actually I didn't realize this but I do know this game I just never heard of it as Othello and so here they basically describe the problem so they're basically saying that some people believe that these models these have seemingly good performance from memorizing certain statistics and they're just autocompletes and stochastic parrots and there's nothing really intelligent going on there the sort of the Counterpoint to that that the people believe is that they do build some sort of world model they do develop some understanding they do develop some emergent abilities and skills that they were not trained for and yet somehow they learned to do so they think of the board of Othello as the world right that grid 8x8 grade as the world but they don't tell the model that it is an eBay grid they keep that secret and in fact the model has no previous knowledge of the game or its rules all that it sees during training is a series of tokens derived from the game transcripts so as they pointed out here before or rather I guess those back on this page you know the moves might be D3 C5 F6 Etc so basically it's just these collections of numbers and letters that describe where on the board that move is taking place and they do not explicitly train the model to make good moves or to win the game nevertheless the model is able to generate legal AFL moves with high accuracy so in Othello like we said the world consists of the the current board position quick explanation of how the researchers approach this problem they want to figure out if Othello GPT a computer program designed for the game Othello truly understands the current state of the game board understanding the state of the game means knowing where all the black and white discs are on the board and what spaces are empty to do this they use a special tool called a probe imagine a probe as a detective that looks inside Othello gpt's thinking process it examines the thoughts or internal signals of the program and tries to predict specific features like the current state of the game board we train this detective the probe to predict the game's board state by looking at the signals inside Othello GPT after certain moves are made if the probe can accurately predict the state of the game board it means that Othello GPT is actually understanding and representing the game's current situation so one thing that we need to know to understand this paper is what a latent saliency map is to better understand what happens in the hidden layers of the neural networks we can use latent saliency Maps these Maps provide a visual representation of the parts of the neural networks engaged in decision making in these images for example the top row are actual photos and the AI is asked to determine if something is present in those images for example it's asked it it seems someone smiling or someone wearing glasses or if there is a blue square in the image below the image for that particular query is a latent saliency map that the AI used to to determine whether the image has those objects for example to determine if an image has someone smiling it focused on the area around the mouth and cheeks to determine if someone has bushy eyebrows it focused on the area around the eyes and eyebrows to find if there is a blue square if focused on the Blue Square in the image this shows that it's understanding what we want and it's looking in the right place to find it as a counter example there was an instance where a model That was supposed to detect if an image was that of a dog or a wolf started to sometimes mistake the two it was later determined that the model looked for items in the background of the image to determine if the object was a wolf or not images of dogs in a snowy forest for example would be classified as wolves by looking at a latent saliency map of that model we would be able to see that if it was making decisions based on what it found in the background instead of solely looking at the animal and so here's the sort of published Laden salinity Maps so each Supply shows a different game State and the top one prediction by the model is enclosed in the black box so it's all of these ones that's the thing that it thinks is the best move or at least a correct move it's the move it's predicting and so in conclusion our experience provided evidence that Othello gbt maintains a representation of game board states that is the Othello world to produce sequences it was trained on so I wanted to jump here really fast and just kind of like maybe over explain this thing a little bit just in case people are not quite getting what is happening because this is kind of important so here I have a notepad with you know it says E4 E5 D5 F4 Etc and then this so it has I believe 64 total moves this is a game of Othello or river C or whatever reverse whatever you want to call it so they made this into a sentence with each coordinate being a word so just like I would say I went to the store today this is that but in or fellow notation of how the players move so probably goes I think black goes first so it's black white black white Etc until no moves are left so they take this game this sentence and they make a whole book out of it so many many sentences I forget the exact where they're using thousands or Millions whatever a huge number of these games AKA sentences and they feed it into the machine and then we start asking the machine to predict move so let's say we get you know we play a game like this and we ask okay what do you think comes next and goes well I think H4 comes next this is not surprising everyone agrees that it can do this at this point no one's denying its predictive abilities because what it's doing is it's sort of predicting it's using statistics to predict next move and it's getting pretty good at it but where the communities kind of splitting up is that there's part of them they're saying it's just this it's just statistics it's just it's just like a parrot that is just kind of repeating what it's heard and there's nothing else going on beyond that and so the other side of the debate and as you see Andrew Noe and Jeffrey Hinton they talk about this they're also saying yeah it is just statistics but there seems to be some emerging skills and properties that are occurring that really seem like this thing is understanding what it's doing it's building certain mental models that showed that it's deducing that it's inferring some information that we never told it about that we never trained it to do and so when we start probing into it and what we begin to realize that you know let's say we get to right here prediction black E6 I mean this is a fake game I made up it has nothing to do with this but let's say you're over here the next thing it predicts is E6 so it's like this is the next move fine that makes total sense but here's where it gets weird we can start asking it to predict the state of all these other squares on the board and again we never told it that there was a board we never told it there were squares winner explain what the different states were the different states are you know it's either empty or has a black tile on it or it has a white tile on it so but that's what they mean by state is like what what's there you know nothing white or black and so if this was just a statistical prediction machine you know and it spits out E6 is the next move and we're like okay but what's on A1 it would have no idea it would say what do you mean what's on a one right well there there wouldn't be any concept of it thinking that there's something there or something isn't there it might not even realize that this is where it is it might not realize that there's a board like there's it has no way of of knowing this this information was not given to it and yet somehow it is able to visualize it predict where all these other pieces are and not only that when it's trying to make its next move these kind of colors show what squares it's looking at to make those predictions so for example I believe they said that red is high blue is low so the very red squares it means that those are really taken into account in predicting the next move whereas the squares are marked blue not so much so for example let's say it's predicting that the white piece will be placed right here what squares is it looking at to make that decision well it's not this one it doesn't here at all if this is empty or if it's white or if it's red it cares not right it's looking kind of like this these squares to see it's looking at the red squares to see that prediction and so the contribution by what they mean by contribution how much do the squares contribute to its prediction the contribution is higher when changing the internal representation of square makes the prediction prediction less likely what's funny is this reminded me of an episode from the office if you've ever watched that show where Michael the boss who's supposed to be kind of not the smartest person in the room he starts acting a little bit weird and one of the employees Oscar who's one probably one of the smarter characters he has an idea he thinks that the reason that Michael is acting weird is that he's aware that he can receive a certain bonus from the company if he completes some objective that kind of goes against the rest of the employees so he asks them a question that normally Michael would not be able to answer do you know do I know what I think you know no does anyone happen to know what 15 of 4 300 is 645 Michael's a genius right why did you say dollars because that is how my mind works what's 15 of 200 thank you everyone Michael is returning the Surplus so he can keep 15 as a bonus so the idea here is similar here's what we told it here's the information we gave it we didn't tell it about a board we didn't tell it about a square we didn't tell it about tiles or whatever so later when we ask what's the state of this particular square and goes oh there's a black tile on it that means it knows about the tile and the board and the square it figured it out it figured it out from this from just rows and rows and lines and lines of this and nothing else but why is this important like who cares if they're building mental models or not here's a video clip that I think kind of shines some light in this this is Jeffrey Hinton so he's often referred to as the Godfather of AI he was the original person that really double down on neural networks specifically something that resembled the human brain the neural network of the human brain as being sort of the key that's going to unlock Ai and he held on to that belief for many decades when it was not the popular belief like he was going sort of against consensus against you know all the other experts in the field and he held on to that belief and he said no it's going to be neural networks and so it's you know 2023 now and uh we're kind of like yeah the dude was right let's take a look at this because these two guys are some of the brightest Minds in Ai and uh it is important to know what they think so let's dive in and uh my name is Wes Roth thank you for watching yeah I think it's fairly urgent for the researchers to come to a consensus about whether these big chatbots like gpc4 or Bard actually understand what they're saying um there's clearly some people believe they do and some people believe they're just stochastic parrots and so long as we have those difference um we're not going to be able to come to a consensus about dangers and so I think it's sort of urgent from for the research Community to address this issue of whether they understand or not and I think both of us believe they do understand but people we respect a lot like Yan think they don't really understand and it's crucial to resolve this issue and we may not be able to come to success about other issues until we've resolved that issue my view is that I believe the large language models and other large AI models are building a world model or building something that looks a lot like the world model so my guts is that I do believe to the extent this building a world model is conveying learning some understanding of the world and one aspect of this is the idea it's just statistics and we all agree that in some sense it has to be just statistics all these things have is the statistics of their input but many people who think it's just statistics of thinking in terms of things like trigram models or counting co-occurrence frequencies of words and it's not just that we believe that this process of creating features of embeddings and then interactions between features um is actually understanding once you've once you've taken the raw data of symbol streams and you can now predict the next symbol not by things like trigrams but by huge numbers of features interacting in very complicated ways to predict the features of the next word and from that make a prediction about the probabilities of next words the point is that is understanding at least I believe that is understanding I believe that's what our brains are doing too that they're not just stochastic parrots