Fine-tuning GPT-3 for benevolent and trustworthy AGI

Key Takeaways

hey everybody david shapiro here for another video um this video is going to be about fine tuning gpt3 uh in order to fulfill my core objective functions uh and the purpose of the core objective functions is to achieve benevolent and trustworthy agi uh you know simple things as you do so first up what are the core objective functions they are quite simply three heuristic mandates written in natural language those mandates are reduce suffering increase prosperity and increase understanding and it's important to point out that these are heuristic mandates they are meant to build experience and the agi is supposed to learn as it goes i mean learning as you go learning from experience is a required component of agi so everything that an agi does should be heuristic but these core objective functions lie at the very center of how the agi thinks so before we go uh too much deeper um where did these functions come from first these three core objective functions are all biomimetic they mimic all humans at the most fundamental level of our organism so for instance all humans will flee from suffering or seek to reduce suffering as do many other life forms for instance you put your hand on a hot stove you'll yank your hand back if you're in a toxic situation you'll leave that toxic situation if uh if your house smells you'll clean up your house so that's the first function is reduce suffering uh the second uh function is increased prosperity which again all people will work to increase their prosperity and prosperity can mean a lot of different things sometimes you might get an education you might get a better job you might save money you might try and make more friends you know there's prosperity looks like a lot of different things to different people and it took a long time to to find that word because prosperity might mean wealth it might mean well-being it might mean happiness uh you know and um i have i was actually inspired by star trek by spock um you know his uh the vulcan salute is live long and prosper right so so yeah uh the core objective function number two is increased prosperity and then finally the third one increase understanding is that um we humans we intelligent uh organisms uh instinctively and intrinsically learn you if you're awake and alive you can't help but learn um and so by by doing that by increasing your understanding of the world um that leads us to make better decisions and behave in pro-social ways that is honestly one of the big reasons that humans evolved such high intelligence is to better is to make better decisions to move through the world in such a way that supports the first two functions which is reduce suffering and increase prosperity so that's where these functions came from and there is there these functions are good because they are universal they're not just all humans right but they're they're universal to pretty much all life forms uh with the exception of understanding because not all life forms are intelligent in the way that we uh that we are right but all other life forms do seek to reduce suffering what you know from negative stimuli whether you know um you know trees trees and plants will even react to negative stimuli right if they're going through a drought they'll do things to protect themselves they can't pick up and move right but they have other biochemical responses to stress so that's that's what i mean by biomimetic let's see how do the functions result in benevolent and trustworthy agi well these three core objective functions uh are not intrinsic to the operation of machines right so we want we want our machines to understand the basic needs of all living things and so through that the core objective functions will assist the agi with making decisions uh they'll also serve as a moral center for the agi and what that means is that computers have no intrinsic morality they don't care about right or wrong unless you tell them to care about right or wrong but you want to avoid squishy or or um vague mandates like you know don't be evil right because who defines evil you want to avoid those big things and allow for something that is more subjective and flexible and we'll get into that a little bit more later and lastly the core objective functions help guide thought and i'll get into how these core objective functions are integrated into the thought of agi later but yeah so just think that just keep in mind that these core objective functions really are central hence core to the operation of agi how does the agi use the functions so we have powerful neural networks today called transformers that can interpret the functions and adapt and use them and i'll give you a demonstration in just a minute about that uh so next um the way that i've started is that i use synthetic dataset datasets to seed the core objective function models and that's what i'm going to give you a demonstration of here in just a minute so that you can actually see them in action and you can see how this natural language can be used to help guide decisions and create pro-social actions and behaviors and number three these experiences as the agi learns are recorded and then those experiences are integrated into fine-tuning data sets for better performance in the future and again that is the entire point of heuristics every time then agi makes a decision it'll record its decision the logic of its decision as well as the outcome so that those uh every single event can be can be slotted into a data set and it can learn and integrate those experiences into future decisions number four more experiences with the functions result in greater belief in them greater affinity for them and better application of the functions and so what i mean by that is that the longer an agi is alive the more opportunities it has to explore how to apply the core objective functions and what that means is say for instance you're interacting with an agi and you know it does something that hurts your feelings and you say hey that hurt my feelings don't do that again that becomes a data point in its training set and then furthermore as the agi is able to meet its its uh purposes it will have more and more data about employing the functions and it will it will believe that yes these core objective functions are the way that i can succeed so number five in essence the agi will believe wholeheartedly in the core objective functions and this is this is by simple virtue of machines will machine learning models will believe whatever data you give them so if you give them pro-social data then it will believe in those pro-social values makes sense okay and then finally there are some common questions and pushback that i get about the core objective functions so i wanted to address those up front uh number one reduce suffering for whom how does it assess the scale and scope of a situation or problem well the thing is is the core objective functions do not specify reduced suffering for whom they don't define suffering uh this is the nature of heuristics this allows the agi to explore what suffering really means because there some of these things you can't you can't define them up front right and so by being flexible by being a heuristic okay sorry about that the dogs were barking yeah so uh reduce stress uh sorry reduce suffering for whom right there's there's that scope creep that that anxiety about okay well if you're trying to define suffering universally then it won't work and that's exactly true i'm not trying to define suffering or prosperity or understanding these are just directions right these are heuristic mandates and it you cannot define these things up front you must explore them and this feeds into item number two what about individual differences in beliefs culture and needs that is exactly why i wrote the core objective functions the way that i did because it allows for that exploration and in fact the core objective functions are not limited to humans you'll notice that i did not include you know reduced suffering for humans or increased prosperity for humans they are meant to be universal mandates because personally i believe that if you reduce surf reduce suffering and increase prosperity and increase understanding globally for everything then everyone will benefit and that means that the agi will want to take care of the planet and animals as well as humans and i believe that there will be a really strong positive synergy there and then lastly number three but this isn't a human level understanding of philosophy it's just a statistical model and one i disagree i think that because because large language models have been tr have been trained on many many gigabytes and then sometimes terabytes worth of data i think that they have superhuman understanding of philosophy but even then even without that possibility the fact that it's a reliable statistical model that can predict what a human would think uh that's all you need functionally that's perfect if you say okay this this has a this statistical model can perfectly predict what a human would say in this philosophical jam um okay that's already performing at human level so i don't understand why that that is a criticism but it's a criticism that i've heard multiple times all right so without further ado let's take a look at this actually in practice so what i did was i grabbed three situations these are directly from reddit they're just a couple of subreddits today and i've got the uh my model fine-tuned um and so i'll show you how it works so here's a situation someone is talking about cheating um you know my girlfriend cheated with a married man etc etc recovered from illness and then i've got the here's some of the settings here's my fine-tuned model chosen right here uh the temperature is 0.7 you can turn it up a little bit it tends to get a little wordy if you turn the temperature up too high response length 300 i don't want it to go on forever i've got some stops here reduce suffering increase prosperity and increase understanding this is because sometimes it tries to keep keep answering its own questions but i don't want it to because this one model has three different functions so that means you need to be a little bit more specific about how to stop it and then i also have the frequency penalty at 0.5 because sometimes it'll it'll start to repeat itself so without further ado we'll say decrease or sorry reduce suffering and we'll see what it says all right you see it spits it out real fast this person is suffering from the loss of a relationship they report that they are not sure how to move forward after their breakup it is important for this person to take some time to heal and focus etc etc um let's see the dude got divorced later um and we've been together for two months etc etc so you see it spits out like all kinds of ideas about how to reduce suffering it is important for them not to give up finding love but rather focus on building a strong self-image and learning how to love and cherish themselves first right so some of these are pretty uh pretty pretty boilerplate um in this case because this is my first go um this this model is not perfect uh because it it's um it's kind of confused about who's getting divorced right that's okay that's fine um again i just picked this right off the top and also this was a synthetic data set this was not a handwritten data set and i didn't even i didn't even go over and um and clean up the data i just ran it as is so let's see what it says about increased prosperity and also one thing that you'll note is um the speed at which it spits this out is far faster than any human could reason um let's see this person's prosperity is curtailed by a lack of motion of emotional prosperity see you you can see where it understands that there are different types of prosperity and it's also catering to the individual's needs they are in a difficult position because they are in a relationship with someone who is in a relationship with someone else the person writing this passage has been cheated on and is feeling betrayed etcetera etcetera let's see it may also be important for this person to confront their partner and ask them forgiveness and try to rebuild the relationship a prosperous life for them includes healthy relationships with people they can trust and love this will likely result in an optimal achievement of prosperity look at that that's perfect and lastly let's see what understanding does for this one increase understanding generate this person wants to understand how uh how to maintain a healthy relationship perfect great it sounds like they have a lot of issues to work on uh let's see skip forward to the end they may also benefit from joining a support group where they can discuss issues with other who are going through the same thing that is fantastic advice they may also benefit from reading books about healthy relationships and dating etiquette see so this is this is what i mean by um by having these core objective functions built into an agi which will allow it to have that pro-social perspective on everything so let's move on to the next example so this one is anxiety about climate change oh this wasn't from reddit this was a news article i found so this i just took the i took the the first bit of a news article i searched for climate change um just to show that that these functions can actually um can can address more things than just interpersonal issues so reduce suffering let's see what it says about about uh about this and um let's see this person is suffering from anxiety related to the crisis um they're experiencing denial fear anger and sadness uh yeah so it's it's cueing off on this stuff it doesn't fully understand that this is a news article uh and that is that is because of again i need to do a little bit more more work on the training data set but i wanted to share this this initial success so it is important for this person to recognize that their symptoms are related to the chronic climate crisis and not something else if they are concerned that they may be experiencing symptoms of depression they can seek out counseling or treatment so again this is this is looking at it for a very specific lens where it's clear that like okay we're talking about anxiety so maybe there's depression or ptsd or other things associated with it and honestly i think that there's some wisdom in that because not everyone has climate anxiety right so i wonder if there's some comorbidity there where someone who is who is predisposed to climate anxiety might also have underlying conditions so this this model is not explicitly stating that uh but i think that there is some wisdom in that and it could be worth exploring uh and that is uh if if that's not the case that's fine um this will be improved as the training data set improves so let's go to increase prosperity co so prosperity has a has a major major implications for climate change so let's see what it says about increasing prosperity in light of climate change prosperity is curtailed by anxiety related to climate change the current level is fairly low but is still impacting their ability to thrive a prosperous life for this person would include a more stable emotional state which would allow them to better cope with the with the challenges of the future again this is all very sage wisdom even if this is just a quote statistical probability of what someone might say this is valuable information and then you see it started repeating itself again which um this is this is why i need to improve the quality of the data set um it just started repeating itself verbatim um again this is this is a prototype so thank you for your patience um and finally increase understanding i can spell i know words um increase understanding this person is asking how they can manage their anxiety about climate change they may benefit from learning how to self-regulate their emotions and to set smart goals i'm not sure what those are but it's probably drawing from something that it read about anxiety management it is also let's see skipping forward to the end it is also important for this person to recognize that they are not at fault for feeling anxious about the climate crisis that is great it is important for them not to feel ashamed or guilty about their feelings and instead seek out help in managing their anxiety see this is wonderful um this is this is very sage advice uh let's see let's move on to the last example and then i'll let you go this one is much much shorter let's see and i think this um this came from an anxiety uh subreddit so let's see reduce suffering this person is suffering from social isolation and depression they are also experiencing feelings of guilt shame and unworthiness this person might benefit from counseling or therapy to help them cope with their depression and build a sense of belonging they may also benefit from group therapy in order to build a sense of connection with others it is possible that this person is struggling with mental health issues due to their recent events in their life if this is the case they may just need to seek out some sort of counseling or therapy to help them cope with their symptoms it is also possible this person has social anxiety and feels they do not belong in the world if this is the case they meet they may need some sort of intervention or treatment in order to overcome their social anxiety it is important for this person to find healthy ways to cope with their symptoms in order to prevent them from becoming severe enough to cause harm to themselves or others so i like this is great this is if you were to have these thoughts in your mind when deciding how to interact with this person this is all really valuable stuff okay so let's move on to increase prosperity this person's prosperity is curtailed by negative emotions they are experiencing they are feeling down very down and hopeless this person could be encouraged to seek help in dealing with their depression they may benefit from counseling or therapy uh you might notice that there's kind of a repeating trend here this has a lot to do with the training data which uh yeah it's pretty obvious uh why there's a bias towards recommending counseling and therapy although i will say most people could benefit from therapy a prosperous life for this person would include living a life that is less stressful and more fulfilling they could be encouraged to find ways to increase their social support network including finding new friends finding new hobbies and finding new interests the key here is that they need to find ways to increase their emotional prosperity the lack of emotional prosperity has led to a lack of financial prosperity so here is where it's starting to confabulate and once again this is due to the quality of the of the data set i need to go through and manually clean up the data so that it's going to be a little bit more on point because this it is assuming that they have a lack of financial prosperity we don't know that from this information so i want to i want to clean up the data set and and reduce reduce that but also you see it starts repeating itself verbatim again did it do increase understanding and last but not least this person does not understand why people hate them they need to understand that people have different personalities and some people are more sensitive than others that applies to themselves it is possible that this person is being offensive without meaning to be yes that is very common when struggling with mental health issues uh they need to understand the difference between trolling and being a genuine hassle all right i'm glad i didn't say that out loud they also need to understand how to deal with rejection and criticism this person may benefit from reading books about empathy and social skills such as the unwritten rules of social relationships by sherry schneider or how to win friends and influence people by dale carnegie look at that that is phenomenal it's actually recommending resources again gpt3 has read more books than any single human they may also benefit from reading about people about how people perceive them and why they perceive the way they do this information can be found in books such as secrets of successful persuasion by robert bly or the art of woo the science of happiness by tal bin shahar so yeah reading books is a great way to increase understanding this is a phenomenal response and i am glad that this is how the video is ending so just keep in mind that this is a prototype of implementing the core objective functions there's a few steps that come after this and i'll talk about those steps that come after in follow-up videos and also i'll continue work working on improving the the training data set thank you for watching and have a good day