Jack Clark — Building Trustworthy AI Systems

Key Takeaways

the challenge is like well I didn't sign up for this like I wanted to do AI research I didn't want to do like AI research plus societal epics and gear politics that's also not my expertise I think that's a very reasonable point unfortunately there isn't like another crap team of people hiding behind sample two entirely shouldering the burden of this you're listening to gradient descent a show where we learn about making machine learning models work in the real world I'm your host Lukas be well Jack Clark is the strategy and communications director at open AI before that he was the world's only neural network reporter at Bloomberg he's also one of the smartest people I know thinking about policy and AI and ethics I really am excited to talk to him I feel like I typically get nervous when people ask me like you know kind of big policy questions about AI and I never really feel like I have much smart to say and I think the goal of this podcast is mainly to talk about you know people doing AI and production but then when I started writing down questions I want to ask you I was like wait a sec hit like I want to ask you all the policy questions and all the weird questions that that ever it asked me cuz I have no idea and I like why should I seriously want to know because I feel like you think about this a lot I mean this is such a cliche question but I'm like actually fascinated by how you're gonna answer it which is what what probability do you put on like AI apocalypse oh good so we'll start with the really easy question and go from there yeah yeah yeah what's your like is it like 1 in 10 like 9 out of 10 one-in-a-million like what do you say I apocalypses is quite high it's like 50% but that's only because the chance of most apocalypse is if they get to the point that they're happening like say pandemic which is something that we're currently going through it's quite clear that most of today's governments don't have capacity or capabilities to these really hard challenges so if you end up in some scenario where you've got like large autonomous broken machines doing bad stuff then I think your chance of being able to like right the chip broke the ship is like relatively low and you don't have like a super positive sort of outlook I think the chance that we have to like avert that and get ahead of it it's actually quite hard but I think your question is more like if something like wakes up and we enter this very very weird territory what are our chances and I think if we don't do anything today but our chances are extremely poor no okay so yeah I think maybe I agree our chance of surviving in AI apocalypse probably low but I think my question actually is is what do you think the tests are like actually entering the AI apocalypse I'd remember that all apocalypse scenarios you know they they kid some some more than one right so like I mean in a way like the like a pandemic apocalypse like unless you think they're sort of like linked that should make the AI pocket right I think there's this is kind of like at the beginning of when you started to do massive amounts of computer trading on the stock market say what's the chance we're going to enter into a high frequency trading apocalypse and I think I'll someone would have answered that is it's really high we'll have problems but it's fairly unlikely but the whole system will topple over due to high frequency trading and I think for my answer on AI is pretty similar like it's really high but we're gonna have some problems because it's a massively scaled technology that does weird things really really really quickly and we'll do stuff in areas where we're finance is also deployed huge amounts of capital so the opportunity for big things going wrong is Caterpie the amount of a total apocalypse feels a little fantastical to me and that's partly because I think for a real apocalypse like a really bad severe one you need the ability for AI to take a lot of actions in the world which means you need robotics and robotics is you and I both know is terrible and actually protects us from the huge amounts of many parts and for like ornate apocalypse and areas I way that I think about this is you develop a load of radical technology and and some of the greatest risks you face aren't with technology deciding of its own volition to do that stuff that very rarely happens it's even unlikely here there's a chance that you kind of get black mold but technology like somewhere in your house you have not been cleaning it efficiently in you don't have good systems in place and something problematic starts developing in a kind of emergent way barely notice but that thing has really adverse effects on you and it's also hard to diagnose the source of the problem with why it's interesting so but okay so that's actually like a little bit less of a that seems like a much more concrete scenario like I guess what dumb what form might that take I mean it sounds like you're mostly worried about sort of the things we're doing now we get we get better at doing these bad things and that causes big problem or like top of mind is like concerns for you yeah well I I guess I bring my concern is we're currently pretty blind to most of the places this could show up and we kind of need something that looks a lot like weather forecasting and you know radar and sensors for for looking at evolutions and mr. main the sorts of things that I'd be worried about are scaled up versions of what we already have recommendation systems pushing people towards kind of increasingly odds areas of sort of content or subject matter but we kept on realizing all quietly radicalizing people or making people behave differently with each other I worry quite a lot about sort of AI capabilities interact with economics so you have some economic incentive today to create entertaining disinformation on this information I think we think about what happens when when those things collide you've got good AI tools for creating this equation or disinformation and an economic incentive and start start showing up I think we're going to be relatively few grand evil plans I think we were going to be lots and lots of like accidental screw-ups that happened at really really large scales and that happened really really quickly we'll self-reinforcing cycles and I think that that's the challenges is you not only need to spot something that you're going to need to take actions quite quickly and that's something but we're traditionally just really really bad at doing as people are we good we can observe bad things happening but our ability to act against them is quite low but yeah I mean so you do like a lot of work on like ethics nai and a lot of kind of thinking about it but it sort of seems like those scenarios sort of feel is like is AI special they're like it seems like there's kind of a lot that might be like just sort of general technology risk right do you think AI makes it sort of different I think delegation so like technology allows us to delegate certain things technology up until many sort of practical forms of AI lets us delegate highly specific things so we can write down in a in a sort of procedural way and AI allows us to delegate things which have a bit more inherent freedom in how you choose to approach the problem that's being delegated to you like you know make sure people watch more videos on my website it's kind of a fuzzy problem you're giving a larger space for the system to sort of think about it and so I think the epics now they're they're not some people humans haven't encountered before but it's full of ethics which is kind of has a lot in common with military or how you do administrative states in the old days which is a sort of ethical nature of giving someone the ability to delegate increasingly broad tasks to hundreds or thousands of capable people that's like a classic ethical problem that people have dealt with for hundreds or thousands of years but with AI now almost everyone gets to be about delegation and that really hasn't happened before we haven't had this scale of delegation and this ease with which people can kind of scale themselves up and so lots of the ethical challenges like okay people now have much greater capabilities to do like good and harm that they did before they have these automated tools that kind of extend their ability to do this how do you think about the role of the tool developer in that context because sure you're building just like iterations on previous tools that the scope of which arose tools will be used the areas in which will be used is much much broader and you've sort of dealt with before and I think it introduces ethical considerations for you but maybe governments OB previously that word I see so so in your view AI kind of allows like single individuals to to have sort of broader impact you know and therefore the tools they actually make available to folks there so there's more ethical issues within that yeah like a good way to think about this is oh I think language models are interesting here's an ethical challenge but I find interesting with language models you have a big language model that has a generative capability you want to give that basically everyone because it's a sort of analogous to a new form of paintbrush it's it's very general people are going to do all kinds of stuff with it except this paintbrush reflects the implicit biases of the system sort of data that it was trained on at massive scale so okay it's like a slightly racist paintbrush the problem is now different to just having a paintbrush you've got like a paintbrush that has slight tendencies and some of these tendencies seem fine to you but some of the tendencies seem to reflect things that many people have a strong moral view of as being like bad and in society what do you do then and I've actually spoken to lots of lots of artists about bearson most artists will just say give him a paintbrush like I won't be like crazy funhouse mirror version of society so I can talk about to make interesting things that feels fine but then they wonder about what happens if if someone gets given his paintbrush and they just want to write checks for a kind of economic purpose they may not know much about the paintbrush they've been given they may not know about straights and then suddenly they kind of unwittingly creating massive scaled up versions of papaya seeds and herring - that thing you gave up but that seems challenging and we're like we used a technology developers have a lot of choice a sort of uncomfortable amount of choice and a lot of problems which are not easy to like fix but you can't fix this you need to sort of bigger I haven't talked about it kimete people aware of it well it's a really clever analogy I have not not heard that one before yeah I mean I think I think it's eat some weird scalability of a lot of this stuff like if we just have tools but let people scale themselves in various directions and the directions are increasingly creative areas because we're building these you know scaled up curve fitting systems that can fit really weird curves simply to get like interesting semantic debate but all the problems of like curve fitting now become we have problems of like a production of parts and sort of force which feels different and challenging I don't have great answers here I have more like oh dear this is interesting and feels like different but actually I mean it's interesting because the like the you know you speak of this sort of like language model is like you know just for example like what if you had a language model but I mean like open a actually like had this issue you know and I'm curious like how you thought about at the time and how you reflect on that now I think at the time so this is GPT - which is a language model will be announced and didn't initially release that subsequent be released in full at the time we I think we made a kind of classic error which is that if you're developing a technology you see all of its potential very very clearly and you don't just see the technology you're holding at your hands you see gen 2 3 4 & 5 and the implications there off I think we treated some of our worries about who misuse of this technology we were thinking about later versions of the technology before what we were actually holding because what actually happened is we release it we observed a huge amount of positive uses and really surprising ones like this game AI dungeon where a language model becomes a kind of Dungeon Master and it feels like interesting and actually different like a different form of game playing something we wouldn't have expected and the misuses were relatively small and it's actually because it's really hard to make a misuse of a technology it's probably as hard to make a misuse of the technology as it is to make a positive news and luckily most people want to do the positive uses so your your amount of people doing abyss uses it's a lot smaller I think that means that the responsibility of technology developers is going to be more about maybe you're still going to kind of trickle things out in stages but you're ultimately gonna go to kind of release lots of stuff in some form it's about thinking about how you can control some elements of the technology while making other parts accessible but can you control how you'd expect a big generative system to be used while making it maximally accessible because you definitely don't want like a big generative model that may have biased tendencies providing generations to people in say a mock interview process that happens before they speak to a human for an interview stage because that's the sort of usage but we can imagine and feels like the sort of thing you really want to avoid but you can sort of imagine ways in which you've made this technology really really really broadly accessible while finally ways to carve out parts where you as a developer kind of say this this probably isn't ok so I because our thinking's become a lot more subtle and I think we did we didn't anchor on the future more than the present and that's been one of the main things that's changed interesting so knowing that you know now you wouldn't withhold the model I think you'd still do staged release but I think that you do role research earlier on characterizing of biases of the model and potential malicious users because I think what we did is we did some of this research and then we did a lot more after some of the initial models to be released on characterizing subsequent models we are planning to release what I think is now more helpful is if you you have a load of that stop front loaded so you're basically saying here's the context here of a traits of this fig which is like going to slowly be released that you should be aware of it and so yeah I think we would have done stuff slightly differently and I think that this is what we're trying to do here is is learn how to behave with these technologies and some of that is about making yourself like more culpable with is traditional for its outcomes because it's a thinking exercise it makes you think about different things to do so I'm glad but part of the goal of GPT to is bring a problem that we actually don't get to get wrong in the future earlier in time to a point where we can like do different ways of release and you know maybe some that'll be good and some will be suboptimal and learn because I think in five six seven years these sorts of capabilities will need to be treated in a sort of like standardized way we thought about carefully and getting to that requires lots of experiments now it's kind of interesting I guess there's sort of two kinds of problems again I think my understanding of the worry with GPZ 2 is actually malicious uses which like more information probably wouldn't help with but then there's also I think like you know your idea of like accidentally racist paintbrush you know like that sort of speaks to like inadvertently bad uses I mean both seem like potential issues but do you now view malicious uses as kind of less of an issue because I really could imagine like a very good language model having plenty of malicious uses I suppose you could say well any interesting technology probably has malicious uses so should we never release like any kind of tool like how do you think about that yeah again it's good what we're doing really easy question a couple of things one of the things we did with GPT too was we release detector system which was a model trained to detect outputs and GPT two models we also released a big data set of unsupervised generations from the model so other people could build different detector systems I think that a huge amount of deal with misuse is just giving yourself awareness you know like why why are police forces around the world and security services able to actually deal with organized crime or we can't make organized crime go away to socio economic phenomenon but they can like tool up on very specific ways to detect patterns of organized crime and I think it's similar here where you need to release tools that can help others detect the output for things you're releasing for avoiding malicious users I think it's actually kind of challenging I think that it's a little unclear today how you completely rule that stuff out I think it's generally challenging to do that with sort of technologies some of how we'd be approaching it is trying to make prototypes the idea being if we can make like a prototype new space that's malicious and real then we should sort of talk to affected people the extent to which we would publicize that remains deeply unclear to me because as you've kind of been sort of interior if you publicize malicious users it's like look over here yes how you might miss universe state we've released which seems seems a little dangerous I think that we're going to need new forms of control of technology in general at some point I don't think that's like this year's problem or next year's problem but you know in 2025 you're going to have these like embarrassingly capable cognitive services which can be made available to large numbers of people and I think sort of cloud providers and governments and other are going to need to work together to really characterize what can be just generically available for everyone and what needs some level of like care and attention paid to it and getting to that's going to be incredibly unpleasant and difficult but it feels part of a noticeable but I guess just to be concrete like if you created sailing at GPT three that was much more powerful you think that you would probably release it along with the detector would be the sort of compromise over I think you think about different ways that you can release it because like some capabilities might find some might you know you might want to have some sort of control so you control the model over people access sort of services around there that could be one way you do it another way it could be just releasing fine-tuned versions of models on specific datasets or specific areas because if you find you know model it's kind of like murals silly putty where you take this big blob of like capability you printed on you dataset it takes on some of the traits of that data set and in some sense you've restricted it so you can do things like that I mean the challenge for a lot of developers going forward is going to be in how to deal with the route like artifacts themselves like the models themselves like here's a thing I think about regularly is it's it's not today it's not next year it's probably not even 2022 but but definitely by like 2025 we're gonna have conditional video models like someone in the AI community or some group of people are gonna develop research but allows us to generate a model generate a video that runs for some period of time you know a few seconds probably probably not like minutes but they can guide it so it it includes specific people and they do specific things and maybe you also get audio as well that capability is obviously something but it's like a much harder case of just a language model or just an every 12 I think with that people he definitely gets like quite a few controls applied to it and needs systems for like authentication of real content on Republic Internet as well like it provokes questions about that yeah I think we're heading we're heading into a weird era for all of this stuff I just I think the advantages you get up releasing all of yourself just sort of publicly of the Internet pretty huge but I also paid for this like to some degree a dereliction of duty by the AI community to not think about the implications of where we are in three four or five years because I have high confidence that we can't be in this state of affairs where the norm is to like put everything online instantly because I think I people just develop things that are frankly like too capable by we I mean in AI research is pretty large bu to be able to do that and say this is fine do you think we're young I need to ask you what is the responsibility of sort of technologists and how do we get to a responsible place necessary and then you could ask me another question for that I don't know it's funny I feel like I really want to want to reserve the right to change my mind on some of this stuff like I feel like yeah I think I think I'm kind of reluctant to like say things publicly because I you know the it seems like actually the ethics really depend on sort of the the specifics of the you know how the technology works and stuff and I think like you know I think like on GPT too is like for just as an example it seemed like you know I thought open a decision was intriguing and like different than I think what I would have done or what my instincts would have been but it was kind of like provocative to say hey we're not gonna you know release this model and I think you know I think the good thing about it maybe it was it kind of got everyone kind of like talking and and thinking about it I guess also another thing that I don't really have a strong point of view on but it was like little interesting is it seems like every it seems like at the moment every AI researcher is sort of asked to be like their own kind of ethicist you know on this stuff like I see like a lot of like you know Ethics documents coming out with you know like even like open source you know ml projects will sort of have like their their code of conduct and on one hand it seems a little it seems a little almost like highfalutin to me like I feel like I have this instinctive like come on like you know like you know should I like put out in like a code of ethics with like you know like the toaster that I sell or you know it seems a little there's something seems a little like unappealing about it but I can actually also definitely articulate the other side of it that if you think I guess to me like it's less like the the power of an individual or and more of just like sort of like if this technology can kind of compound and like you know run amok then you know maybe it's a case that you know people really should be thinking about it but yeah it's honestly I don't know and I don't even know I guess I'm curious what you think about this cuz you're like in this all the time do you think that AI researchers are in the best position to decide the stuff I mean if it if it really affects society's profoundly as you're saying it seems like kind of everyone should should get a say about how this stuff works right yeah so this is unfit right what's actually happening here is an unfair feed for AI researchers which is that they are building powerful technologies they're release them into a world that doesn't have any real notion of technology governments because it hasn't really been developed yet and their release women to systems but we'll use the technologies to do great amounts of goods and and maybe a small amount apart and so the challenge is like well I didn't sign up for this like I wanted to do AI research I didn't want to do like AI research plus societal epics and geopolitics that's also not my expertise I think that's a very reasonable point unfortunately there isn't like another crack team of people hiding behind some bull two entirely sholden bergna fists there are ethicists and social scientists and philosophers members of republic governments all of them have thoughts about person should be involved but I think the way to view AI researchers is they're making stuff that's kind of important they should view themselves as being analogous to engineers of like the people who build buildings Mitchell bridges don't fall over you have a notion of ethics chemists you have a notion of ethics because chemists get trained how to make bombs and so you kind of want your chemists to have a strong ethical compass so that most of them don't make explosives because until you have a really really resilient and stable society you don't want lots of people able to be really have sort of no ethical grounding because they might do experiments that lead to literal blows or you know people like lawyers who have codes of conduct in their base it's very strange to look at AI research and sort of more broadly computer science and see a relative lack of this when you see it in other disciplines that are as impactful or maybe even less impactful on our current world and I don't think any a young century is going to solve this on their own but I think for the culture of culpability of thinking but actually to some extent I am like a little responsible here not a lot it's not my entire problem but I have some responsibility is good because how you get systemic change is you know millions of people making very small decisions of their own lives it's not like millions of people making huge of optional decisions because that doesn't happen at scale but millions of people making like slight filters is how you get massive change over time I think that's kind of what we need here hi we'd love to take a moment to tell you guys about weeks and biases weights and biases is a tool that helps you track and visualize every detail of your machine learning models we help you debug your machine learning models in real time collaborate is 'le and advanced the state of the art in machine learning you can integrate weights and biases into your models with just a few lines of code with hyper parameter sweeps you can find the best set of hyper parameters for your models automatically you can also track and compare how many GPU resources your models are using with one line of code you can visualize model predictions in form of images videos audio philately charts molecular data segmentation maps and 3d point outs you can save everything you need to reproduce your models days weeks or even months after training finally with reports you can make your models come alive reports are like blog posts in which your readers can interact with your model metrics and predictions reports serve as a centralized repository of metrics predictions hyper parameters trade and accompanying notes all of this together gives you a bird's-eye view of your machine learning work though you can use reports to share your model insights keep your team on the same page and collaborate effectively remotely I'll leave a link in the show notes below to help you get started and now let's get back to the episode well let me ask you another easy question what do you think about military applications of AI I think that well the military applications of AI on special in the sense that it's technology but it's going to be used kind of generically to different domains it'll get used in military applications I mostly don't like it because of some of what I think of is there like a p-47 problem so you know the ak-47 was a technological innovation to make this type of rifle like more repeatable more maintainable and easier to used by people who had much less knowledge of weaponry than many prior systems you developed this system it goes everywhere it makes the act of like taking life carrying out war cheaper and more repeatable massively cheaper and much more repeatable and so we see a rise in in conflict and we also see that this artifact this technical artifact to some extent like driest conflict it doesn't create the conditions for conflict but it gets injected into them and it create an it worsens because it's cheap and it works and I think that AI if applied sort of wrongly or rationally in a military context does a lot of this it makes certain things cheaper certain things more repeatable and seems really really bad I think AI for military awareness is much more of a kind of gray area like lots of some ways in which unsteady piece sort of holds in the world is by different sides you your award each other having lots of awareness of each other awareness of troop movements distributions what you're doing and they use surveillance technologies to do this and I think you can make a credible argument that the the advances in computer vision but we're seeing that's being applies like massively widely may if if adopted at scale by lots of militaries at the same time which is kind of what seems to be happening may provide some some diminishment from a certain type of conflicts because it means there's generally more awareness I think stuff like the moral question of lethal autonomous weapons is really really challenging because we want it to be a moral question but it's also mately going to be an economic question like it's going to be a question but governments make decisions about on the motivation of like economic speed and decision and what it does the strategic advantage which means it's really hard to reason about because neither you or I make these decisions and actually accommodate with like a radically different frame probably of like a strong intuitive push against from it existing but that's not the frame of these people right right let's do is oh my ass dude what else you got Lexi okay this is maybe like a less um a less loaded question but I'm cool I'm actually like genuinely curious about this so you know you recently put up this paper I think it's called towards trustworthy AI development and I thought the you know as someone who builds a system that does a lot of saving of experiments and models and things like I thought is really intriguing that you picked as like the subtitle mechanisms for supporting verifiable claims so it seems like you draw this incredibly bright like direct line between you know trustworthy AI development and supporting verifiable claims and I was wondering if you can sort of tell me why that that that is so connected well it's it's really easy for us to savings that have immoral or unethical kind of value and in words committed organization to something like we we value you know the safety of our systems and we value them not making you know biased decisions or or what have you but that's an aspiration and it's very similar to a politician on the election campaign trail being like well if you elect me I will do such-and-such video I'll give you as money or are like I'll build this thing but it's not very verifiable like you're sort of needing to believe the organization or believe a politician and they can't get much proved to you because a is going to be really really significant in society that's going to play an increasingly large role people are going to approach it with slightly more skepticism just as they do with anything else of their life but plays are like large role and aspects of them and they're going to want systems of recourse systems of diagnosis systems of sort of awareness about it now today for most of this we just pull back on people we fall back on like the court system you know as a ways you'd like insure stuff very viable we have these mechanisms in the law that mean that if I as a company make a certain claim you know especially one that has a fiduciary component the the sort of validation of that plane comes for loan and stuff around my company and the ability to verify it comes from action and also legal recourse if I'm not doing it tons of stuff like that but I guess like what but just before they like you like this because some people will not have at the paper listening this silly when you say like supporting verifiable claims like what's an example of like a claim that you might want to verify that would be relevant to trust for the area development is that say our system is we feel that we've like identified sort of menu for main fire season our system and have labeled it as such however you know we we want the world's sort of validate but our system lacks lies in the critical area so we're going to use mechanism a bias bounty to get people to compete to try and find biased rates in our system and survey you've got to be you're making a claim about it I believe that it's you know relatively unbiased or I've taken steps to long for bias in it but then you're introducing an additional thing which is a sort of transparent mechanism for other people to go poke holes in your system and find bias easily and that's going to make your claim more verifiable over time and if it turns out that your system had something like huge trading cattle swatted well at least four mechanism helps you identify too many various rate from there similarly we think about the creation of like third-party auditing organizations right so basically you could have an additional step you could have I have a system making some claim about bias putting a biased bounty out there so I have more people like hitting my system but if I'm being deployed in a in a critical area and what I mean by critical is you know a system that makes decisions that affect someone's financial life so you know any any of these areas where policymakers really really care about then they can say okay my system will be audited by a third party when it gets used in these areas and so now like I'm really not asking you to - believe me I'm asking you to believe like the results by public County and the results of this third party auditor and I think when all of this stuff kind of stacks on itself and gives us the ability to have to have trust in systems other things might be I will just you know I will make a claim about how I value see but the mechanism by which I will be trading my models and aggregating data will be using sort of encrypted machine learning techniques so there I've got this claim but you can really verify it because I have an auditable system that shows you kinda sort of preserving your privacy while manipulating your data and so the idea of this report is basically producer loaded mechanisms but we and a bunch of other organizations that people think are quite good and then the goal over the next year or two is to have organizations who are involved in the reports and obviously weren't implement some of these mechanisms and try them out and we'll be trying to do this with oh so I can join the red team - yeah I think like so obviously having we recommend a shared red team that takes a little bit of unpacking because obviously if your two proprietary companies your red team's can't share lots and lots of information about your proprietary products but they can share the methods they use - like Red Team AI systems and making standardized on some of those sort of best practices that kind of thing feels really really useful because eventually you're going to want to make claims with your red team the system and it's going to be easier to make a trustworthy claim if you use a kind of industry standard set of techniques that are well documented that many have done but if you just sort of cowboy it and doing yourself so yeah please join the red team we want lots of people on like some shared red team infrastructure eventually but the red team infrastructure is actually it seems like the way you describe it and I'm sure this comes from security but I just I'm not super familiar the field it's like you have someone like internal to organization right like you we have an internal team that that tries to break or tries to find problems with you have that and then you're seeking to find ways to have your internal team share insights with other people at other organizations now they can't say here's of a proprietary system I broke and what I did but they can say when I like to sit down and crack my knuckles and try and like red team an analysis that here the approaches I use spective we not in red teaming but we had actually done a little bit of this and open the eye we're in a GPT to preserve people we wrote about some of the ways we try to probe the model for biases because we think that this is an area that's generally useful to especially useful to get standards on and then since then we have just been emailing our methodology to like lots of people at other companies these people can't tell us about the models from their testing providers but they can look at their like probes we're suggesting and tell us if they seem sensible and so that shows you how we're like able to develop some shared knowledge without without breaking sort of proprietary stuff interesting do one thing I kept kind of thinking is as as I was as I was reading your paper is like I use all kinds of technology that I don't think has made verifiable claims like I mean I feel like I rely on you know all kinds of things to work you know and maybe they're making claims but I'm certainly like not aware oh well I sort of assumed that internet security works I assume that you know I now have like all these things plug into my home network that could yeah but I just sort of what do you think that it sort of seemed like these might be just sort of best practices for developing any kind of technology or do you think there's something like really AI specific within it and where would you even like draw the line where you would sort of call something yeah that sort of needs this kind of treatment I think some of it comes down to when when you draw a line I think a I saw is basically when you cross through a technology but can easily be sort of altered and analyzed and half the scope of its behavior to find to a technology where you can somewhat orde it and analyze it and sort of list out where it'll do well but you can't fully define its scope and I think that a lot of like just sort of once you train in your on that you have this like big like probabilistic system but will mostly do certain things but it actually has a surface area that's inherently hard to you characterize fully it's very very very difficult to like fully list it out and mostly it doesn't make a huge amount of sense to you because only a sort of subset of the area of the service area of your system is actually going to be used at any one time so it does have some kind of differences or you know bias counties right is a kind of weird thing it's sort of equivalent to saying all right before we elect this like mayor or before we appoint this person to an administrative position we want a load of people to us from a ton of different questions about quite abstract values that they may or may not have because we want to feel confident that they reflect the values what we'd like someone to have in that position that feels different actually it feels a little different like normal technologies and it would be observed to expect we get to a world where everyone verifies it replay they make all the time because you have the time you know I mostly go through my life depending on on my own belief that other people are sticking to the rules of the game but we all have some cases where we want to go in on something that's happening in her life and oiled it every single facet of this and I think the way to think about why you need verifiable claims or ability to make from quite broadly is as government's consider how to sort of govern technology and how to let technology do the most good while minimizing their the harm it's probably going to come down to the ability to verify certain things in certain critical situations so you're kind of building a little bit stuff not for the majority of your life we pull the really narrow edge cases where this has to happen but necessarily that means you need to build quite general tools for verification and then try and apply it with specific areas it's interesting that why don't it seems like there's been a lot of sort of complaining about AI research recently that a lot of the just the research claims which are maybe not so loaded and not so apply to we don't interact with are actually not really verifiable yeah I mean some of these things are just because there is a computer gap there is like a minority of organized large amounts of compute varies a majority of organizations and a huge swath of academia if not all of academia but has very real computational limits and this means been at a really high level you can't really validate claims made by a subset of the industry because they're doing experiments at scales which you can't hope to meet so some of this is about what one of really general tools we can create just resolve some of these kind of a symmetries of information because some issues of verifiability or less about your ability to verify specific thing at that moment it's more about having enough kind of cultural understanding of where the other person is coming from that you kind of understand what they're saying of a premise behind it and can trust them which is less you demanding a certain type of verification but being like okay well you know you're a complete alien to me you come from another cultural context or another you know political ideology however we have this sort of strong shared understanding of this one thing but you're trying to get me to believe you about and right now if like certain organizations wanted to motivate academia to a certain type of research it would depend on I come from this like big compute premise land and I'm asking you to hear me when I list out a concern that only really makes sense if you've done like experimentation of my scale because that's calibrate my intuitions so we need to find a way to give these people the ability to have the same conversation so that you can sort of improve that so are you gonna give them a ton of compute like what's your participation we basically specifically recommend for Bay but governments fund cloud computing which is a bit difference - it's a bit wonky right but well one thing you need to bear in mind is that today a lot of the way academic funding works sort of centers usually on the notion of having some bit of hardware or capital equipment that you're buying and as we know like that stuff depreciates faster than cars it's like we're bye you're a researcher at an academic institution you'd be much better place to buy like a cloud computing sort of credit or system system that lets you access a variety of different clouds work generally when we go and work with government pushing this idea but they should fund some kind of credit that backs onto a bunch of different cloud systems because you don't want the government saying all right all of America is gonna run on like amazon's cloud but it's obviously like a bad idea but you can probably create a credit which backs on to the infrastructures of like final safety large cloud entities and deal--but requested concerned family and I think this is surprisingly tractable it's like some some policy ideas are relatively simple because they don't need to be any more complicated and so we're kind of lobbying the lack of a better word governments to do this I think the other things bear in mind is that lots of governments because they've invested in supercomputers really want to use supercomputers as their computer for academia and that mostly doesn't work you actually mostly need a dumber simpler form of hardware for most forms of experimentation so you're also saying to government's like I know you spent all of this money on this supercomputer and it's wonderful and yet it's great at simulation you kill our weapons whether you don't need it for Miss stop trying to use it for this like exclusively so that's also about some nice encounter though that's an interesting feel like we've spent untold billions on like having the winner of the top 500 list and we're in some pitched geopolitical war with China like of course you want to use this for AI and you're like yeah but look some people just want like an ATP server actually most people are fine with that so you and this big is not like easy to like multiplex and sample out to people compared to like AWS or Microsoft or interesting well we so we're a little bit running out of time and I asked you I'm curious but we always end with two questions there I'm particularly just it in your point of view on this so yeah the first one I mean and you actually you really view a lot of things going on today I I mean from your vantage point at open air and then also the newsletter that you put out so what would you say is like the topic that people don't pay enough attention to you like the the thing that like you know it's just matters so much more than people compared to how much people look at it I think the thing that no one looks at for really matters is advances in just a very niche politic computer vision which is the problem of re-identification of an object or a person that you've seen previously and what I mean is that our ability to do pedestrian reai densification now is is improving significant it's stacked on all of these image net sort of innovations in steps or more ability to do rapid like feature extraction from video feeds it's stacked on like a load of just interesting components innovations and it's creating this stream of technologies that will lead to really really cheap surveillance but eventually is deplorable on edge systems like drones or whatever by anyone and I think what we're kind of massively under estimating the effects of that capability because it's not that interesting it's not an advanced it doesn't even require like massively complex like reinforcement learning or any of these things that researchers like to spend time on it's just a sort of basic component but notice the component that supports surveillance states and authoritarianism and balance the component but can make it very easy for an otherwise sort of liberal government to slip into a form of surveillance and control that no one would really want - ha and I'm actually thinking about yeah can I write like a survey or something about this because it's not helpful for someone like open AI to warn of this it's sort of a wrong message it's may be okay for me to write about a kekkaishi of my newsletter as I do but I sort of think about writing an essay like has anyone noticed this if I gather all of the scores right I look at all of the graphs and stuff and speak by folder of it it's all going up like it's all great hockey stick it's all getting cheap yeah so that's a cheerful but I think it's important yeah well great answer as expected all right it's a good question which we always ask and normally we're we're talking to kind of more industry practitioners but maybe can apply it to open air so when you look at the demo projects that you've witnessed and like Oprah has actually had some really spectacular ones um what's the like what's the part of sort of like conception to to complete that looks the hardest here and maybe them the most unexpectedly difficult piece of it like sort of watching you know like solving dota or being there's even a donor like GP to like what like where do things get stuck and and why good question um I think there may be two parts where projects get like stopped all have interesting traits one is just data like they used to really want visa to not matter so much and then you just look at it and realize that you know whether it's like dota and how you ingest data from like the game engine there or robotics and how you choose to do like two main randomization in simulation or supervised learning where you're just figuring out what data sets they have and what what mixing proportion do I give them during trading and how many buttons so I do that just seems very hard I think others have talked about this it's not really a well-documented science it's something that many people treat with intuition and it just seems like an easy place to get stuck and then the other is testing once I have a system how well can I to rise it and and what sort of tests can I use from the existing research literature and what tests do I need to sort of build myself like we we spend a lot of time to figure out new evaluations at open AI because for some systems you want to do a form of eval that doesn't yet exist to characterize performance in some domain and figuring out how to test for a performance trait that may not even be present in the system is really hard so most of you two areas yeah okay I can't help myself actually as you're attacking I I find him like one more question they watch I'm sorry to do this but I've won so like I feel like the people that I know or that I've like watched closely at opening have been actually spectacularly successful and and like you know they've been part of projects that have really seems to me have succeeded like the the robot hand doing the Rubik's Cube and dota are they're like a whole bunch of products that are a project that we don't see that I've just totally failed universe that was sort of a failure we tried to like we tried to build the system which is kind of like opening I gin but the environments would be every flash of HTML game but have been published on the internet and they said yeah so that failed right that failed because of network asynchronicity and so basically you ended up having because we were sandbox and the things in the browser and you had a separate game engine with you to go and talk about the network to them all rail actually isn't really robust enough to that level of like time jitter to do useful stuff so that kind of didn't work and so we have some public failures which are because it's kind of yeah we have so some kind of private ones a lot of it is you know some people just spend a year or two on an idea but then it's not not working out some people and I won't name the project as its public but maybe they came up with a simple thing but worked really well and we spent six months trying to come up with what as a researcher before it was the board discipline more like better approach to it and the simple thing was work all of these they tried to eventually published your system with like a simple thing I know like yeah it works but I don't know you but rather let my complex idea works um but we don't like out big bets like a hand or don't or all GPT those intended to go okay and that's usually because they've come from a place of iteration like dota came from prior work applying PBO and I think evolutionary algorithms to two other systems the hand came from prior work on just like block rotation right so once you can do block rotation you can do a remix TPT came from prior work on scaling up language models just of GPT one so a lot of it's just happened sort of iteratively of a public generic but yet we don't have like we don't have an abnormal lack of failure nor an abnormal amount of success like because it's pretty pretty in distribution I know okay yeah yeah thanks all right [Music]