Yes. There’s a real phenomenon to describe, even if it’s difficult to pin down.

Over the last thirty years, seventy-seven Nobel Prizes in Chemistry have been awarded to humans, and zero to chimpanzees. An alien, on hearing about this fact for the first time, might wonder if the Nobel Committee is biased. But no, there really is something going on with humans that sets us apart from chimpanzees.

It’s an overly obvious point, but obvious points can sometimes matter. There are abilities we possess that let us walk on the moon, and that put the fate of the planet into our hands rather than into the hands of chimpanzees. Philosophers and scientists can debate the true nature of intelligence, but no matter what they conclude, the underlying phenomenon remains. Something about humans has let us achieve feats never before seen in nature; and that something has to do with our brains, and how we use them to comprehend and affect the world around us.

The fact that we can’t give a precise definition doesn’t mean that it can’t hurt us.

If you’re caught in a forest fire, it doesn’t matter whether or not you understand the underlying chemistry. You burn all the same.

The same is true for intelligence. If machines start converting the surface of the Earth into their own infrastructure, while generating so much waste heat that they boil the oceans, then it won’t much matter whether we have a precise definition of “intelligence” yet. We’d die all the same.

We mean this literally, and we’ll be exploring why we expect such extreme outcomes from smarter-than-human AI over the coming chapters. In Chapter 3, we’ll argue that superintelligent machines would pursue ends. In Chapter 4, we’ll argue that those ends would not be what any human intended or asked for. Chapter 5 is where we argue that their pursuits would be better accomplished if they took resources we were using to survive. And Chapter 6 is where we argue that they’d be capable of developing their own infrastructure and rapidly turning the world uninhabitable.

You don’t need a precise definition of intelligence to build intelligence.

Humans were able to create fire before they understood the underlying chemistry of combustion. Similarly, humans are well on their way to creating intelligent machines, despite their lack of understanding — as we’ll cover in Chapter 2.

Rather than thinking of intelligence as a mathematical notion in need of a precise definition, we recommend thinking of “intelligence” as the label for an observed natural phenomenon that we don’t yet understand well.

Something about human brains allows us to perform an astonishing variety of feats. We build particle accelerators; we develop new pharmaceutical drugs; we invent agriculture; we write novels; we execute military campaigns. Something about human brains means that we can do all of those things, while mice and chimpanzees can do none of them. Even if we don’t yet have a full scientific understanding of that mental difference, it’s useful to have a label for it.

Similarly, it’s useful to be able to talk about intelligence that surpasses our own. We can already observe AIs today that are superhuman in a variety of narrow domains — modern chess AIs, for example, are superhuman in the domain of chess. It’s natural to then ask what will happen when we build AIs that are superhuman at the tasks of scientific discovery, technological development, social manipulation, or strategic planning. And it’s natural to ask what will happen when we build AIs that outperform humans in all domains.

If and when AI shows up that can do world-class scientific research thousands of times faster than the best human scientists, we may protest that it’s “not truly intelligent,” perhaps because it reaches conclusions in a very different way than a human would. That could even be true, depending on what definition of “intelligence” you choose. But the real-world impact of the AI will be enormous, however we choose to label it.

We need some terminology for talking about that sort of impact, and for talking about the sorts of machines that are radically capable at predicting and steering the world. In this book, we take the easy route of assigning the label “intelligence” to the capabilities, rather than to specific internal processes that give rise to those capabilities.