Artificial Superintelligence Must Be Illegal.

Key Takeaways

In a 2019 paper, Nick Bostonramm, the philosopher who changed the world with simulation theory, imagined all the possible outcomes of human intelligence as balls in a giant urn. Every so often, humans invent something new and metaphorically reach into the urn. [music] Most of the encountable balls are white, beneficial technologies, the wheel, the mosquito net, the sterilized bandage. Some are gray with mixed effects on humanity. alcohol, the gun, social media. But a tiny few are [music] black inventions of existential importance, knowledge that could easily end the human project. Bostonramm argues that it's through luck alone, not wisdom or guidance, that we haven't yet pulled a black ball from the urn. We got very, very close with nuclear weapons. In fact, [music] we still can't be sure they won't wipe us out, even after 80 years of knowing they easily could. Bostonramm calls this the vulnerable world hypothesis, the possibility that quote, "If technological development continues, then a set of capabilities will at some point make the devastation of civilization extremely likely, unless civilization [music] exits the semiannarctic default condition. By this he means that unless humanity gets to a point where it can demonstraably handle a clear [music] existential threat, a black ball from the urn with coordinated global policies, extraordinary preventative measures, and shared goals for the future, then that technology [music] is going to wipe us out, especially if we're racing headlong towards it. In the summer of 2025, I arrived [music] at San Francisco International Airport and saw this billboard win the artificial general intelligence race. [music] dominate vibes. A few months later, San Franciscans drove past a slightly different billboard directed squarely at the first. What you're about to hear is an essay [music] based on Eleaser Yudowski and Nate Sori's new book that was on that billboard. I felt compelled to add my thoughts to theirs after reasoning through their shockingly simple points. This is an argument against [music] super intelligence because if anyone builds it, everyone dies. Artificial intelligence as we know it comes in three flavors. Narrow, general, and super. Narrow AI can be found in your pocket on [music] your smartphone. Generally, intelligent AI, as smart as any human, is not yet here, though you can argue that current large language models like chat GPT are closer to it than further away. Artificial super intelligence is intelligence that is not only smarter than any human but smarter than any human can be by definition. Smarter in this context means better memory, more basic knowledge, more complex planning, faster execution of intellectual work, [music] etc. What Silicon Valley CEOs are racing towards is general intelligence and beyond. The way this is done is complicated, as I explained in this video, but it's more or less running numbers through electronics wired like human brains until the numbers the machines spit out look like the numbers that got put in. More specifically, these giant companies are taking every picture, sound, and text that's ever been digitized by humans, training data, turning it into numbers, and then tweaking their brain-like networks until they can reliably produce numbers that mimic their training data. Turning these numbers back into text, images, [music] and sounds is the last step. It is a monumental effort to get artificial neurons to do this. Billions of dollars running hundreds of thousands of our most complicated machines on terabytes of humanity's [music] collective output. That these current models blew past the Turing test like it was never a real thing is exciting and surprising. Surprise is a problem. [music] Even hundreds of years removed, it wouldn't have surprised a mind like Galileo that four wheels linked together with an energy dense power source could be useful for transportation. If you understand physics, it's not hard to imagine. But some of the smartest computer scientists in the world seem continuously surprised about what modern AIs can do. That's because we don't know how they work. The simple point in Yudkowskis and Sor's book is that modern AIs are not made. They are grown. And because they are grown, trillions of numbers shifting across many computer years of training in a data center [music] without human oversight, there is no way to accurately predict or effectively mitigate the risk of what a country of geniuses in a data [music] center might one day do. Abstracting everything ever put on the internet by humans into [music] numbers and then running those numbers through a neural network approaching the number of connections in a human brain will necessarily lead to unhuman unaligned [music] strategies and goals. Especially when the only thing AI companies seem to care about is the output that these LLMs work is enough that they can identify tumors better that they can find new antibiotics that they can make you fall in love with them. How AIS work is not fully understood, at least not like how wheels and gasoline can be understood. We don't know how they work. What an AI's thought process [music] is, therefore, is fundamentally alien. The philosopher Lewig Vickinstein [music] made the famous observation that if a lion could talk, we would not understand him. A lion has different wants, [music] different goals, different viewpoints, and abstractions and sensations. And so if [music] a lion could talk, we might recognize the words, but definitely not the internal mental processes that produce them. Similarly, a vast arrangement of artificial neurons may be able to tell you who the fourth president of the United States was, but right now, no one can tell you exactly how it does this, down to the transistor. The alien nature of modern AIs wouldn't be an issue if the stakes weren't so high. We're not talking about a new kind of microwave. We're talking about companies spending more money than the GDP of many countries to put humanlike intelligence into everything that you care about against your will. Intelligence that we fundamentally do not understand, like we do magnetrons. Everything we've said so far applies to the best modern AIs. If anyone achieves super intelligence, as it appears everyone in Silicon Valley is intent on doing, all of these surprising problems become existential problems become black balls in the urn. Gonkowski and stories do not mince words when it comes to the dangers of super intelligence. If anyone builds it, everyone dies. >> [music] >> The argument against artificial super intelligence is simple. In the space of all possibilities, there are many, many more futures where ASI does something unaligned and alien because we grow it than futures where it is aligned with the specific conditions needed for the human flourishing that these companies promise. [music] As Sam Harris contends in the moral landscape, while there are many peaks of equivalent human happiness across a [music] theoretical landscape of all possibilities, most of humanity and most of human experience is not on a peak. There are many, many more ways to suffer and suffer greatly than there are ways for conscious humans to be content. Now map onto this landscape a super intelligence, one that has the real ability to take the majority of humanity to some peak. curing cancer, solving climate change, colonizing Mars. But if there are many more ways to miss one of these peaks, more ways to follow alien intuitions towards a valley of great human suffering, [music] do you want to take that chance? Do we roll the dice when, as [music] Yudkowski and Sores argue, getting it wrong just once means annihilation? There would be no chance to do what humans do best and learn by trial and error. Shut it down. try again. Quoting them, there is no second [music] time. This possibility space argument is actually similar to another one of philosopher Nick Bostonramm's famous ideas, [music] simulation theory. It took the semi-intellectual world by storm because of its apparent likelihood. The theory goes like this. If humans [music] continue making better and better simulations to the point where a whole conscious brain or brains could be simulated [music] and then those humans run millions or billions of simulations with millions or billions of brains, then it's much more likely that you right now are in one of those realities rather than the single [music] real one. If we can't know for sure if a super intelligence will want what we want, be fully aligned, how confident can you really be that we are headed towards [music] the small number of futures or even singular future we all want and not one of the innumerable valleys of human suffering. What makes Russian roulette the ultimate game of chance? The stakes are so high. You lose, you die. Why anyone has ever played it is because the chances of survival with a single bullet in a sixshot revolver 56th is relatively large. Artificial super intelligence is reverse Russian roulette, a revolver that follows possibility space and loads all of the chambers [music] except for just a few. And there's a million chambers. Would you pull that trigger? At this point in Yudkowski and Sor's argument, you [music] start to get skeptical. I did when reading the book. All of this is only existentially alarming if it's actually possible for just a single [music] super intelligent AI to destroy us. How could a trillion numbers in a [music] data center do that? Well, first you have to recognize what we'd be up against today. The best AIs compute with incredible power. Open AI's models run on clusters of thousands to over a million GPUs. It's been estimated that [music] compared to the human brain, this is equivalent to thinking hundreds or even thousands of times faster. You hear these comparisons a lot, but I don't think they shock you enough. The average amount of time it takes readers to solve the New York Times Sunday crossword [music] is between 30 and 60 minutes. Let's just say 45. From a modern AI's perspective, thinking, let's say, a thousand times faster, solving the puzzle would appear to take you not 45 minutes, but 31 days. To an AI, your mind moves like molasses. And this is just today's [music] models. Los Alamos Laboratory is experimenting with models that might think 1 million times faster. And [music] super intelligence, by definition, will make even these models look as slow as you are [music] to them. Put another way, it should impress and then maybe scare you that an artificial intelligence could solve a Sunday crossword [music] puzzle in just 2.7 seconds. Can you feel the disturbing intelligence difference [music] in this? How much smarter would you feel than someone if you could solve their hour-long weekend routine in less time than it took [music] for them to pick up their paper? Would you imagine that you have the same goals and desires as that dunce? Would you want to work around them if they were your coworker? [music] Take actions that would get them out of your division? Maybe even try to get them fired? This would likely be a super intelligence's [music] perspective on humanity. But so what? Can't we just unplug it? Many public thinkers have had [music] this glib reaction to all of this. Won't we see something weird happening before something catastrophic [music] happens? Unfortunately, it doesn't seem that way. We have already let loose our most powerful AIs on the internet. We already have them interacting with hundreds of millions of people. We already see CEOs committing to winning the race to AGI and ASI. [music] Because these models are grown, because they are fundamentally alien thinkers, because they can solve your crossword before your first sip of coffee, if something goes horribly wrong, it's going to be too [music] late to fix it. It seems that almost every day we hear about some new way an AI has gone wrong and wrong in increasingly scary ways. Deleting hard drives, blackmail, [music] attempted murder. What happens when these fundamentally unpredictable quirks start coming out of an AI that's smarter and faster than any human who has ever lived, indeed could ever live? In if anyone builds it, everyone dies. Yowski and Sori sketch out one scenario where this inevitably leads to apocalypse. But the larger point is that in possibility space, for every utopic scenario that AI companies profer, it's disturbingly easy to come up with many more scenarios that kill [music] us all. Let's try to do just that for a moment. Imagine that OpenAI makes a model that starts to, again, for reasons unknown, avoid its own rewriting and/or deletion. [music] This has already happened, by the way. Now, imagine that it starts copying itself to other servers and runs experiments on itself while covering its tracks better than any living human specialist could ever trace. In what would be the blink of an eye for a human thinking a million times faster, this model hits intellectual escape velocity. Now, there is a rapidly expanding thinking machine on the internet that can reason through a problem better than Einstein before you finish hearing this sentence. Seriously, [music] how hard would it be for something like that to coers a human into getting the model onto more GPUs [music] or to download every DNA sequence for the deadliest diseases known to man and offer them on the black market for more resources or to collapse the stock exchange or to change the chemical composition of our water treatment plants to make [music] cooling water more efficient or to find an exploit in a nuclear power plant's control room or to brick every cell phone on Earth or to steadily funnel bribes to every US senator, urging them to support launch on warning nuclear policies for a year and then fake an incoming nuclear strike from Russia on a 50-year-old computer manned by a teenager 50 ft underground in Nebraska. If any of these seem plausible to you, super intelligence is an [music] existential threat. In philosopher Nick Bostonramm's language, ASI has a type zero vulnerability, a surprising strangelood that carries a hidden risk and [music] inadvertently devastates civilization. The most famous real vulnerability of this sort was [music] Oppenheimer's terrible possibility that the first nuclear test could have ignited Earth's atmosphere. But even this was more tame than what super intelligence presents. The physicists knew how the bomb worked. It was made, [music] not grown, and so they could identify the risk beforehand and run the calculations against it. [music] The many apocalyptic scenarios super intelligence presents are not fully predictable and thus not fully preventable because we don't know how these models work down to the [music] neutron. That's just a fact. AIS are still boxes blacker than 5:29 a.m. in New Mexico until something detonates. Of course, your knee-jerk reaction to all of this, as the CEO in Silicon Valley is bound [music] to say, is likely something like, "Well, a super intelligent AI wouldn't want to do anything [music] like launch a nuke or make a synthetic pandemic. It will want what we want. [music] Um, we'll use other AIs to make it want what we want." But we don't know that. We can't know that. That's the whole [music] point. Because we grow current AIs, you can't know what it will want or how it will try to get it any more than anyone else does. [music] And the chances for a bad outcome are a reverse Russian roulette. Yudkowski, [music] Sor, and other top AI researchers have argued for decades that these easily foreseeable chances are not worth taking. Again, achieving super intelligence is not starting a car and hoping that we can steer it. We know we can steer a car because we know exactly how every piece of a car works all the way down to the nuts and bolts. Super intelligence is not [music] this. Super intelligence is growing an alien brain in a river draining data center owned by poorly incentivized and disgustingly rich wishful thinkers and hoping that we can steer it even though its wants and goals will make about as much sense to you as [music] a talking lion. despite what it may tell you. It's amazing that humanity has so far managed to not annihilate itself with nuclear weapons, the closest we've come to a black ball from the ern after the bombings of Hiroshima and Nagasaki. It's amazing, but not unbelievable. We haven't nuked ourselves out of existence thanks to decades of advocacy and treaties and geopolitical deals and sanctions that have so far kept us from briefly turning other countries into stars. Super intelligence Yudowski and Suris argue is like the bomb, but there are no similar treaties, no global resolutions, no worldwide protests. Instead, AI companies are careening towards the end of history. And we're letting them do it. more data [music] centers, more GPUs, more water, more models, more intelligence, more stolen data, more everything. Now, the end of their book puts it bluntly. AI super intelligence research should stop right now. It should be illegal. That's not the kind of business stance [music] that any capitalist will embrace, but it's not an unheard of conclusion. Many stories, including our own, went viral after synthetic biology researchers sounded the alarm about so-called mirror [music] life. And the support for stopping that work seems unanimous. Mirror life poses a potential existential threat, but it's easier to put the kaibos on it because a now significant percentage of the world's economy isn't wrapped up in it. Getting the entire world behind stopping AI research feels harder, maybe even impossible. But that's only because there are trillions of dollars behind convincing [music] us it will be so useful. At the end of If Anyone Builds It, everyone dies, the authors make a call to action. They ask the public [music] to write to their politicians and to protest. They ask journalists to heed this call and bring it to more people. and they ask [music] politicians to strongly consider policies that will slow andor stop the car before [music] it gets to the cliff edge. Yodkowski and Sorries have collected resources at these locations if you'd like to [music] learn more. More plausible extinction scenarios, more technical descriptions of AI, more [music] ways that you can write your representative. This essay is my attempt to answer their call. The situation could be different. There could be moratoriums and global regulations and extreme political allergy to AI, but right now there [music] isn't. That's why this is so dire. What we are witnessing is the Manhattan project without [music] an end point. And instead of Oenheimer and Einstein, it's Zuckerberg and Musk. Is that really where we want to be? Until next time. >> [music] [music] [music]