How will we know when we've invented AGI? How will we know it is complete?

Key Takeaways

hey everybody david shapiro here um someone asked a question on the open ai forum uh after i posted my recent artificial cognition cognitive entity video they asked how will we know when we've achieved agi how will we know when that's complete and the the first answer the short version is um it'll never be complete um like us it will keep evolving and keep learning forever but a more nuanced and complicated answer is that uh it'll be very really difficult to know and the reason is because once something is smarter than us it'll be really hard to understand it um so for instance any anyone who's a who was a gifted child and is now a gifted adult like myself will be familiar with the idea that um we kind of have to dumb ourselves down to reach a general audience um so like if you're really smart you understand things that other people don't and so like if you talk at your level other people are going to be lost and so what you have to do is you have to kind of simplify what you're talking about in order to be understood and so as soon as agi um these intelligent machines comprehend more than we do they are going to have to simplify and explain to us what their reasoning is what their logic is uh so that'll be that'll be one tipping point but if something is beyond our comprehension then how do you know that it's going in the right direction and so this leads to another critical thing which is why i work on my core objective functions or the heuristic imperatives is because we will want to ensure that before agi becomes incomprehensible because it's too smart and it has to dump itself down for us to understand it we're gonna need to ensure that um that it is both self-correcting and self-improving um so self-improvement is just okay i did this thing let me do it better next time that's not as simple as it sounds because you have to know enough about whatever it is that you're trying to achieve um in order to do it better and your you gotta you have to be able to measure your own performance but self-correction is a whole other ball game so what do i mean what do i mean by self-correction self-correction is okay i'm going to hold myself to a higher standard like say for instance you're out with friends and you mistakenly say something that hurts someone's feelings and you say ooh first thing is you have to identify that you hurt someone's feelings and that it's your fault so you say okay well i want to do better next time i don't want to hurt my friends feelings because then they don't want to hang out with me and so you self-correct you say okay why did i do that now and there's a whole host of cognitive behaviors that go into this you have to go back through your memory what was i thinking at that time what was i feeling at that time why did i do that was it an honest mistake something that i just didn't know better if that's the case then i need to learn more social graces or maybe someone said something and i was irritated and so i lashed out and in which case the answer is i need better emotional self-awareness and self-control so that's what i mean by self-correction but if instead of a person who's going out drinking with some friends you're an agi or an artificial cognitive entity that uh that has you know power over life and death and you know the ability to influence global politics and climate change and stuff how do you then self-correct right how do you measure your success and your your flaws and so the the net result is we need to spend a lot of time working on self-correction and self-improvement before agi becomes incomprehensible because the thing is is like if someone is less intelligent than you and they they don't know what they're talking about you just be like okay you don't make any sense but it's because like you've made x y and z mistake the thing is if someone is more intelligent than you you might make the same mistakes where you say oh well i don't understand what you're talking about therefore you're wrong and stupid right but and in many cases the opposite is true it's like actually i'm the one that's wrong and stupid because i'm talking to someone who knows more than i do so if we get to that point where agi knows more than us and we think that it's wrong and stupid but it's actually smarter than us like how do we know but also if it's beyond our capabilities how do we how do we test it and make sure that it's self-correct so this is the nature of the control problem anyways this video is way longer than i meant for it to be so i just wanted to share these thoughts real quick great question