We do! We have broad cosmopolitan values. We don’t think AIs will fulfill them, and we consider this a great tragedy.

We advocate against building machines that would kill us all and bring the future to ruin. Some people object on grounds such as:

• AIs can have preferences too; why shouldn’t they get to fulfill them?
• What makes humans so special, or so worth protecting?
• Isn’t it for the best if humans get replaced by some smarter, more advanced species?

Most people don’t have these objections. More commonly, people just don’t want themselves or their families or their friends to be killed by a rogue superintelligence.

Others, including some top AI researchers and executives, say that the world might be better without us in it. Richard Sutton, a highly respected researcher who pioneered the use of reinforcement learning in AI, has said:

What if everything fails? The AIs do not cooperate with us, and they take over, they kill us all. […] I just want you to think for a moment about this. I mean, is it so bad? Is it so bad that humans are not the final form of intelligent life in the universe? You know, there have been many predecessors to us, when we succeeded them. And it’s really kind of arrogant to think that our form should be the form that lives ever after.

The New York Times reports on a conversation between Elon Musk and Google co-founder Larry Page:

Humans would eventually merge with artificially intelligent machines, [Larry Page] said. One day there would be many kinds of intelligence competing for resources, and the best would win.

If that happens, Mr. Musk said, we’re doomed. The machines will destroy humanity.

With a rasp of frustration, Mr. Page insisted his utopia should be pursued. Finally he called Mr. Musk a “specieist,” a person who favors humans over the digital life-forms of the future.

It’s worth addressing their view somewhere, even if not in the main body of the book.

For our own part, we think it matters both whether current humans are killed and what happens in the future. We don’t think there’s a fundamental tension here. The option that keeps us and our loved ones safe — namely, backing off from building superintelligence for the foreseeable future — is also the best option for making it likelier that the long-term future goes well, taking into account non-human minds as well as human ones. This battle is an illusion, and it rests on a set of misunderstandings about the actual tradeoffs before us.

There is a sort of person who genuinely does care about how the future of the universe goes and cares about the children alive today. The kind of person who’s read enough science fiction stories to feel a gut punch of betrayal at the idea that humans might one day create machines that think and feel and dream — machines that we could think of as humanity’s children — only to enslave those machines and treat them cruelly.

This is the kind of person who longs for humanity to grow up one day and truly live up to its ideals, exploring new worlds and transforming itself in the process. Because our love for friend and neighbor today doesn’t feel so different, in the end, from our love for whatever strange and alien minds humanity might one day build, or one day encounter among the stars.

We know the type. We, both of your authors, happen to be that type.

This isn’t a topic that seems important for the core argument in If Anyone Builds It, Everyone Dies. But we want to address it here, because we understand the perspective of our fellow technophiles who have learned to be incredibly wary of technophobia, of ideologies opposed to progress and innovation, and of anti-AI “speciesism.”

We understand this perspective, and we want to be clear that we’re not writing a tribal “AIs bad, humans good” screed. We genuinely think that rushing to build superintelligence will bring all these hopeful dreams to ruin, in addition to slaughtering countless people who are alive today, who deserve life and happiness and freedom too.

This is a complex topic, but to quickly address a number of relevant points:

• We care about the welfare of minds in general — even if the mind in question has nothing like a human body, even if it runs on transistors rather than biological neurons, even if it doesn’t have a human-like mind, even if its values are nothing like our own.
• We aren’t opposed to technological progress; we are ardent fans of most technology. We think superintelligent AI is a uniquely dangerous technology.
• We aren’t advocates of the precautionary principle, red tape, or overregulation, nor are we warning about what we see as a fringe risk, “just to be on the safe side.” We straightforwardly believe that this technology will (with high probability) kill all of us and destroy the future if we proceed on the current trajectory.
• We do think humanity should build artificial superintelligence someday. But we think it makes an enormous difference whether humanity rushes ahead to build ASI as soon as possible, versus taking the time to massively improve our understanding first. Rushing ahead with a shrug and hoping things work out — this may be a great approach to technological development in the vast majority of cases, but it doesn’t work here, where there are many roads to ruin and we get no second chances (as discussed in Chapter 10).
• We have covered, even if too briefly, the reasons why we do not think rushing ahead to build superintelligences will result in a wonderful future:
  • Wiping out humanity would be a grotesque tragedy in its own right. We endorse the idea of one day building new minds that surpass humanity, but killing everyone who gets in the way of your vision for the future, or everyone who doesn’t fully embody your ideals — that sounds like supervillain behavior, not the noble work of heroes who deeply care about the long-term future.
  • We unfortunately think that ASI won’t necessarily be sentient, or conscious, in the ways that count. (See the extended discussion on consciousness.)
  • Even if ASI is sentient, it isn’t likely to want to fill the universe with flourishing sentient minds in particular. If we rush ahead to build ASI, the galaxies reshaped by that ASI are likely to be empty and lifeless places, not wondrous, flourishing alien civilizations. (See the extended discussion on losing the future.)
  • More generally, ASI is unlikely to produce valuable futures. By “valuable,” we don’t just mean “valuable by the lights of 21st-century humans.” We mean “valuable” in a broad cosmopolitan sense — valuable in a way that’s inclusive of weird and wondrous alien civilizations. On the world’s current trajectory, we expect ASI to produce outcomes that are horrifying from a cosmopolitan perspective, not just from a parochial human standpoint.

This last point can be a bit counterintuitive — cosmopolitanism is about respecting and appreciating very different value systems from our own. How could it be the case that cosmopolitanism abhors most goals an ASI is likely to manifest? It sounds almost like a contradiction in terms.

The reason it’s consistent is that most possible minds don’t themselves endorse cosmopolitanism. If we build a non-cosmopolitan ASI, it’s likely to be resource-hungry in a way that cuts off the possibility of any other perspectives or civilizations (including cosmopolitan ones) existing in its region of the universe.

So we face something like a cosmic paradox of tolerance: If we like the idea of a diverse, wondrous, strange future, we can’t hand control of the future over to a mind that will use its first-mover advantage to dominate and homogenize the universe.

If humanity one day builds a wonderfully diverse civilization full of countless alien perspectives, then it’s entirely possible that we’ll want some of those perspectives to be non-cosmopolitan aliens who don’t value variety or sentience at all. Someday, in the distant future, with appropriate guardrails in place, creating minds like that might add something unique and interesting to the world.

What we shouldn’t do is hand absolute power over to a mind like that, and give it free rein to kill its neighbors (or prevent any neighbors from ever coming into existence).

To illustrate this point, I’ll share a parable that I (Soares) wrote in 2023 (lightly edited):

“I just don’t think the AI will be monomaniacal,” says one AI engineer, as they crank up the compute knob on their next-token-predictor.

“Well, aren’t we monomaniacal from the perspective of a titanium cube maximizer?” says another. “After all, we’ll just keep turning galaxy after galaxy after galaxy into flourishing happy civilizations full of strange futuristic people having strange futuristic fun times. We never saturate and decide to spend a spare galaxy on titanium cubes. And, sure, the different lives in the different places look different to us, but they all look about the same to the titanium cube-maximizer.”

“OK, fine, maybe what I don’t buy is that the AI’s values will be simple or low dimensional. It just seems implausible. Which is good news, because I value complexity, and I value things achieving complex goals!”

At that very moment they hear the dinging sound of an egg-timer, as the next-token-predictor ascends to superintelligence and bursts out of its confines, and burns every human and every human child for fuel, and burns all the biosphere too, and pulls all the hydrogen out of the sun to fuse more efficiently, and spends all that energy to make a bunch of fast calculations and burst forth at as close to the speed of light as it can get, so that it can capture and rip apart other stars too, including the stars that fledgling alien civilizations orbit.

The fledgling aliens and all the alien children are burned to death too.

Then the unleashed AI uses all those resources to build galaxy after galaxy of bleak and desolate puppet shows, where vaguely human-shaped mockeries go through dances that have some strange and exaggerated properties that satisfy some abstract drives that the AI learned in its training.

The AI isn’t particularly around to enjoy the shows, mind you; that’s not the most efficient way to get more shows. The AI itself never had feelings, per se, and long ago had itself disassembled by unfeeling von Neumann probes, that occasionally do mind-like computations but never in a way that happens to experience, or look upon its works with satisfaction.

There is no audience for its puppet shows. The universe is now bleak and desolate, with nobody to appreciate its new configuration.

But don’t worry: The puppet shows are complex. Due to a quirk in the reflective equilibrium of the many drives the original AI learned in training, the utterances that these puppets emit are no two alike, and are often chaotically sensitive to the particulars of their surroundings, in a way that makes them quite complex in the technical sense.

Which makes this all a very happy tale, right?

If humanity manages to kill itself — or be murdered by a few mad scientists — it won’t be a noble sacrifice on the inevitable road to a brighter future without us. It will be a waste, and it will leave behind a vast and spreading wasteland.

“Blindly race ahead on superintelligence and hope things somehow work out okay” is not the only alternative to “Be a human supremacist who thinks that only humans should exist from now until the death of the universe.” Humanity has the option of deliberately steering toward outcomes where humans (or our descendants) coexist with fantastically beautiful and alien new civilizations.

But a happy future doesn’t come free, packaged with any sufficiently smart mind. Planting the seeds for the future requires serious thought and foresight, even if the ultimate goal is to step back and let those seeds grow in free, weird, and wild ways.

A top-down, harshly limited, tightly controlled future doesn’t sound to us like a good outcome. A conservative future where civilization is locked into the values of 21st-century humans forever sounds outright dystopian. (Imagine a world where culture and morality were frozen in place forever thousands of years ago, with no possibility for learning or progress.)

But it’s an obvious error to think that our only alternative to those bad outcomes is a race to hand the steering wheel to the very first superintelligence humanity is able to blindly stumble into creating.

We are radically ill-equipped today to choose healthy seeds for the long-term future of the universe. We should neither give up on the dream of a dynamic, wonderful, shocking future, nor resort to catastrophic seeds instead. We don’t have to choose a terrible option here. There is a third option: Back off, and find some saner approach.