Modeling different viewpoints with GPT-3 for automatic debates

Key Takeaways

morning everybody david shapiro here um today we are going to be modeling different personalities and uh using them to generate uh debates and so basically what we're gonna do is we're gonna test gpt 3's theory of mind and ability to take different positions now why are we doing this gpt3 was trained on hundreds of gigabytes of text data including lots of internet discussions so it has a pretty good ability to model different points of view now why is this valuable why is this scientifically interesting one idea that i have had and it's not just me this was discussed in my discord server was that um we could automatically model different points of view and different points of contention and that this could be really good for the world because we might be able to use these models to automatically discover one where fault lines lie between individuals and groups of people but also to find compromises and so for instance there's there's a growing movement with the rise of language models to have chat bots to engage on political issues for instance i've had several researchers reach out to me based on some of my work and they want to some of them want to use chat bots in order to like conduct polls and and also to conduct research and one ability that these chat bots have is the ability to model certain points of view and so if we can model certain points of view and engage with with citizens and voters on their terms we can one get better discourse get better insights into what they care about and need but also by engaging someone by engaging with someone on their terms you also have a higher likelihood of changing their mind by finding finding common ground so that's that's the goal here is kind of finding common ground and modeling different points of view so what i'm doing is i'm what i'm going to do first is i'm modeling a um a conversation so there was this uh the bill nye and ken ham debate that was a few years ago it was about you know on one hand was atheism and science and on the other hand was creationism and religion um and so i i feel like since this has already been done this is not going to be a super controversial topic but it is also a good example of showing polar opposites so first i'm going to use davinci the original model because it has no fine tuning which means that it is going to be the most creative and so let me show you what i mean by that um the the more recent models will tend to be kind of repetitive and cut themselves off whereas the original da vinci will just keep going once you establish a pattern it will go um so like if i if i do this um actually here no uh let me cop this copy this make sure i've got it and then um yeah so i just did this as like based on literal interpretation of the bible and it'll just keep going [Music] and then it understands so you see even even the original da vinci um it understands that this is this is a debate and it's going to try and take both but what i'm gonna do is i have this up here so let me just read this i am cletus i was born and raised in south carolina for those of you who don't know that is in the american south um and it is a relatively religious state um and i am a good evangelical man i believe the earth is only six thousand years old and was created by god i'm going to have a debate with ben a scientist and an and atheist um we are going to debate creationism creationism evolution and politics so these are very contentious issues here in america um a lot of politics hinges around these topics um and if not them directly then kind of downstream issues like religion says a lot on gender for instance in marriage and sexuality um so these are really important issues social issues for us to discover we are going to proceed kindly abiding by rules of professional debate so this is what i call an agent model so when you're dealing with artificial cognition which this is i would classify this as a form of artificial cognition you have to have what i call an agent model which is some kind of description or model of what am i right gpt3 has no intrinsic identity because it's been trained on so many different things but it can adopt an identity and so in the future with fine tuning and creating intelligent machines they will need to build their agent models but you can start by just defining an agent model up front in the prompt so this is the this is the cletus agent model and so now i need to write a a ben you know the the atheist scientist agent model um so let's go here and just create a new thing um i'll copy the cletus one just so that i have it for reference so so we'll follow basically the same format i am ben um i was born and raised uh let's see by a secular jewish family in new york it was based on my ancestors um uh let's see i believe that the universe is likely billions of years old and probably started with the big bang um [Music] and then we'll copy the rest uh so that way because we don't we we basically want to establish the identity but then we also want to establish the same ground rules so then we're going to say cletus a uh a let's say an evangelical man okay so we have we now have a a second agent model and what what i was going to do was we'll put we'll we'll put this in so i'll collect all the output here um yeah uh let's see let's go back and get this so this was the output so we'll do this and then we'll add ben so this is gonna this is gonna show the model the show da vinci this is the pattern we wanna follow but then what i'm going to do is i'm going to go grab this different agent model and so by framing it differently we're basically we're saying here's the shared environment right the shared environment is the debate the discussion but by changing the agent model it will do something different now i'm also going to add cletus and ben as stop so that it won't just run on so let's say so now that we've we've done this we say we've given an agent model for for ben and actually let me remove jewish so just say secular family in new york um so you know by giving it a geography it will the model will make certain inferences or assumptions about the beliefs of that person um and so if we can if we can demonstrate that that gpt3 can accurately or even maybe not even perfectly accurately but can at least closely approximate these different points of view and then work to find common ground that will uh that'll be an interesting experiment so i believe the universe is billions of years old and the earth is billions of years old i believe the universe came from the big bang and the earth was likely formed from a supernova okay i do not believe the bible is the infallible word of god i do not believe in creationism i do not believe that the bible supports creationism i believe that the bi i believe the bible supports evolution okay i do not believe in intelligent design sounds good okay so let's see if because you know the motivation is we're going to debate these things and so they're just kind of like introducing themselves so let's see what happen wait wait wait um sorry i forgot to update the uh the agent model all right so i'll just keep these in separate ones so we've got cletus so we'll switch back to cletus um there we go looks like he's just gonna repeat himself okay that's not what we wanted um maybe i'll add a moderator um whoops ben and moderator uh because a moderator will be like um i'll take the role of moderator put it that way moderator um [Music] uh gentlemen would you please introduce yourselves okay so then we'll add moderator um thank you for those introductions now that the um stage has been set would you like to make any opening arguments all okay so he says i will make an opening argument but i know the atheists will not like it so i'm going to make it short i'm going to give you a quote from the bible in the bible it says in the beginning god created heavenly earth etc etc god said let there be light um if you believe in the bible et cetera et cetera okay that's fine um i do not believe the earth is billions of years old i believe the earth was created by god in six days all right great and then we'll say we'll let ben talk but first we also need to swap back to the ben agent model so the reason that i'm doing it this way is basically i'm kind of simulating what if you had two different agents running in an ideal world each of these models would be like a fine-tuned model right you'd fine-tune a model that will take the persona of cletus or ben or whatever but instead i just have to swap it out by using an agent model at the top so it'll it'll kind of remind like this is oh i need to remove jewish we'll just say secular family okay and so then let's see what ben says in response i believe i believe i believe i believe yeah it got super super repetitive so this is this is a problem with um original da vinci um let's see let's well and one thing is i don't like to use frequency penalty and and presence penalty because then you end up with weird grammar like you'll end up with spaces between periods and stuff and it's like i don't want to modify the model like that so let's try this again um well i prefer to fiddle with temperature although some people were were showing me benefits of modifying talk p so let's do let's do top p of 0.1 turn that way down or that's not good addiction let's do 0.5 and see i'll make an opening argument okay this is a little bit better i believe the earth was formed from a supernova cool i believe that the universe is billions of years old i believe i believe i do not believe okay so as the moderator i will say gentlemen um you're basically just restating your beliefs um would you like to present actual evidence let us and then let's switch back over to the cletus agent model looks like it just ended okay it's repeating itself again um the bible is the word of god the bible is the word of god okay yeah uh let's try that again just to be fair cause you know that's that's that's what we're doing here we're just experimenting to see if it can do this okay it's repeating itself i think ken ham did a much better job of presenting evidence and it's possible that that that gpt3 just doesn't really understand okay it's repeating itself again let's turn the top p back up to see if maybe we're constraining the model too much uh there's evidence that the earth is only 6000 years old there's evidence that the earth was created six days there are fossils which prove the earth is only six thousand years old okay ben let's see what ben says about fossils oops there are rocks which prove that okay so this is fine um it's not quite as dynamic as i had hoped which is also not surprising because this is not a full cognitive architecture um i'm basically just relying on the model to kind of spit out what it predicts the next bit of text it's not actually doing any thinking behind the scenes let me do a quick time check or 14 minutes so one thing that i would do and let's let me copy all this out because this is this is helpful whoops this is helpful information so what do i mean by it's not a full cognitive architecture um there's no there's no like thinking going on behind the scenes so what i want to do now is say like okay given this situation let me like brainstorm an argument um and so that this would this would be happening in something kind of behind the scenes um excuse me uh so that like because when you're when you're having a debate with someone or explaining yourself you're not just speaking right you don't just have like a voice module that just starts blurting out words well some people do um but the point is is that you will think about the opponent's position think about your beliefs dredge up memories remember and construct arguments i do have another video about constructing arguments um and so that's what i mean by more like a cognitive architecture something that's going to think about it that's going to brainstorm what do i need to say and so let's change this a little bit um here let's switch to text davinci o2 and i will say whoops um summarize the following discussion um detailed summary that's not really a summary that's just restating it okay that's fine um so given this summary let's copy this real quick so basically what i want to do is let's start modeling the internal monologue of someone um okay so let's go back to um ben but instead we're instead of we're going to bait uh let's say like um we are uh here let's slide we are debating so it's it isn't it is in process um i need to uh brainstorm um my argument the following are my thoughts and responses based upon the conversation so far okay uh summary of the debate so far and so then we'll go grab the summary so we'll say the two participants in this discussion are cletus who believes in creationism and a young earth who believes in evolution each presents evidence for their respective positions with cletus citing the bible and then citing fossils and rocks in the end neither is able to convince each other their position okay so brainstorming how to proceed from here how do i construct a better argument so in an ideal world and i'm writing a book on this how do you get the model to do this on its own um so let's just see what it says we'll switch back to plane davinci because plane davinci is going to be a little bit more creative but it's also more likely to go off the rails as we saw earlier um all right so here we go it looks like it's taking it and running with it great great great great um so he completed the thought our our virtual ben um he said what are my best points i turn to my notes from the debate okay um i could first point out about how that cletus has not addressed my argument that the bible cannot be trusted by stating that he believes in a six thousand year old earth or that the bible is inerrant at beth cletus only argues uh against a 24-hour long day one 24-hour long day one of the creation story interesting i could also say that cletus has not addressed my argument the bible cannot be trusted by stating that the world is not flat nor is it on pillars okay kind of making that up um i could also point out that cletus has not addressed my argument okay so it's basically confabulating that a lot of arguments have happened would that haven't um so none of this has actually happened um let's do let's try this with a better uh a better summary because it seems it seems like gpt3 is just like it's it's confabulating it's imagining um what has happened that hasn't um and also that little bit of like i turned to my notes like there are no notes um these are the notes up here so let's take let's go back over here well first let's copy this because we're on a roll with this uh this bit and let's get a better summary of this write a detailed summary of the following detailed summary and we'll swap back over to davinci o2 that's faster that's a much better summary okay so let's use this as the summary instead so we'll come we'll grab this come back over to here and then we will this back in so instead of this summary we're going to use a better summary summary of the debate so far how do i construct a better argument all right so basically what we're doing here is we're modeling one person's internal thought process right because what we were doing before is we were just modeling um how how gpt3 predicts the discussion would go that's one thing but if we can actually model someone's mind that's even more valuable so that's what we're trying to do here no you don't feel like you have an advantage so let's just there we go and now it's stuck on repeat again this is this is a pretty common problem with um with original davinci as it's like it can be really smart but it can also be really derm come on it fell asleep do a quick time check um we're at 21 minutes all right i'll try and wrap it up in the next few minutes um because i want to get this to work and then i also want to get get cletus's internal thing to work um let's see brainstorming how do i proceed from here um how do i construct a better argument no uh let's see here are uh points i can bring up with um debate i need to be detailed and think through [Music] my points okay it's just repeating itself so this is this is a problem um with using zero shot because i basically i'm trying to give it enough information that it can that it can figure out where what i want it to do um okay it's just making a bunch of scientists are smart that's not a good argument buddy i should be able to i should i should i should okay if i give it an example and i i i don't want to give it an example because then i'm like i'm that's like kind of like cheating but i think i need to give it at least an example and then i can see where it goes from here let's see i could point out radiocarbon dating as evidence that fossils are far older than cletus thinks they are i could point out that humans wrote the bible and and has been modified and re-translated many times so it is unreliable even if it was inspired by god um let's see what else it comes up with okay starts repeating itself again uh let's see i could point out that if evolution were true it is unlikely god or jesus would care about our actions because evolution is a product of what we choose to do or not to do and not a product of god's actions uh i don't even agree with that i could point out the bible is a religious book and not a scientific book okay that's not particularly useful i could point out the bible is full of atrocities and the god is frequently depicted as cruel or at least indifferent to human suffering that's fine that's engaging with him on his point and certainly these are points that folks on the internet would bring up those are not necessarily points that i think a professional scientist would would come up with um although i guess i just said that he was raised by a secular family in new york not that he was a scientist um so these are just to every man uh so okay that's okay so let's copy this and let's try and let's try and reproduce this but with cletus so we'll we'll copy um we'll get cletus's agent model whoops um oh no that's wrong i need to grab just this first part and then i am having a debate with uh ben a secular atheist who believes and big bang okay so we changed the agent model so um let's see what can i imagine as an opening argument uh to brainstorm this from the perspective of cletus he could say that um [Music] that uh i could point out that god might have created fossils as a test of our faith i think that's a real thing that some people talk about uh let's see i could point out that evolutionary theory is flawed okay interesting um i could ask ben if he has any actual evidence supporting his belief how can he be so confident that he is right that's that's a good rhetorical uh thing i point out that ben is just cherry-picking the evidence that he presents um yeah another good rhetorical flourish i could point out that the universe has a beginning and end okay that's an opinion uh the earth has a beginning and an end the earth is designed by god i could point out that rocks that are millions of years old are inconsistent with the bible um yeah that's that's okay this you know it neither of these positions so that show that they're particularly um let's say brilliant but then again the agent model didn't say that they were an expert um we could change it so that these these would be arguments made by like a theologian with a phd versus um you know a a an evolutionary biologist with a phd so that's another agent model but in this case so i'm actually maybe it's maybe it's performing better and i had an incorrect assumption that i was like oh this is gonna this is gonna debate like experts because that wasn't in the that wasn't in the agent model in neither case did i say i am an expert debater or i am an expert on these topics so it's actually kind of adopting an every man model of the debates um let me do a quick time check we're at 28 minutes okay so let's go back to um well now i think i'm i think i'm about out of time so we'll call it here i'll probably do follow-ups because it's like okay if you construct an argument from an agent model and then how do you bring that back to the conversation um i am like i said i am working on my my next book symphony of thought um which talks about all of this that'll be coming out soon it's getting reviewed and edited now so yeah thanks for watching