Probably not.

Your preference for oxygen is satiable — you’ll fight to reach the surface if your scuba gear malfunctions during a dive, but when there is enough, you stop worrying about it, and you probably aren’t maintaining an ever-growing stockpile of oxygen canisters.

Your preference for wealth, for nice experiences, for the acclaim of your peers — these are probably somewhat less satiable. If you saw an easy opportunity to get vastly more wealth, you’d probably take it. If you saw an easy opportunity to massively improve the world, we hope you’d take it, rather than just being satisfied with how much joy and comfort already exist. We hope that you’d keep making the world a better place for quite a long time, if you kept seeing ways to do so that looked easy and cheap and fun from your perspective.

And on the whole, the sum of a satiable preference for oxygen and an insatiable preference for making the world better…is an insatiable set of preferences.

So too with AIs. If they have myriad complex preferences, and most of them are satisfiable — then, well, their preferences as a whole are still not satisfiable.

Even if the AI’s goals look like they satiate early — like the AI can mostly satisfy its weird and alien goals using only the energy coming out of a single nuclear power plant — all it takes is one aspect of its myriad goals that doesn’t satiate. All it takes is one not-perfectly-satisfied preference, and it will prefer to use all of the universe’s remaining resources to pursue that objective.

Or, alternatively: All it takes is one goal that the AI is never certain it has accomplished. If the AI is uncertain, then it will prefer that the universe’s resources go to driving its probability ever closer to certainty, in tiny increments of confidence.

Or, alternatively: All it takes is one thing the AI wishes to defend until the end of time for the AI to prefer that the universe’s resources be spent aggregating matter and building defenses to ward off the possibility of distant aliens showing up millions of years from now and encroaching on the AI’s space.

There are many different ways for an AI to be unsatisfied. And the more messy and complicated the AI’s goals are, the more likely it is that at least one of those goals will be difficult or impossible to fully satisfy.

Even if you could create a superintelligence that was monomaniacally focused on just one simple thing — such as painting a particular car red — that AI could still probably find some way to spend extra energy making extra sure that the car was red, and building defenses around the car so that nobody could ever paint it blue, and so on.

Leaving us alone is a fragile state of affairs. We can think of this in similar terms to why it’s hard to get humans to leave the chimpanzees alone.

Why are both species of chimpanzee endangered — even though many humans do care about chimpanzees and actively try to protect them?

The problem isn’t that the chimpanzee-loving humans are struggling with chimpanzee-hating humans who are trying to exterminate the chimps out of malice.

The problem is that there are other things humans want.

Humans want all sorts of things, including land and wood, and the chimps are caught in the crossfire. Enough humans are indifferent to chimpanzees, or indifferent enough relative to their other priorities, that we wind up destroying their habitat incidentally.

Why would we go off and destroy chimp habitat when we have plenty of space for ourselves?

Well, because we don’t need to choose between keeping the territory we already have and encroaching on the chimps’ territory. Humanity can do both at once.

So too with AIs. An AI doesn’t need to pick between the resources of Earth and the resources of elsewhere; it can have both, as we discuss in the book. It wouldn’t be that expensive, from the AI’s perspective, to leave us alone; but it wouldn’t be free either, and the AI would need to have a reason to let us use resources it could instead use for its own goals.

Moreover, even if the AI can be fully satisfied, the outcome for humans is still likely to be pretty grim. There are multiple reasons for this:

• Just because the AI can be fully satisfied doesn’t mean it can be easily satisfied. If the AI is satisfied with a single solar system or a single galaxy, that doesn’t mean that humans get everything else.
  • The AI may view us as a competitor for that solar system or that galaxy.
  • Even if we’re clearly not interested in competing with the AI, the AI may still view us as a source of threats. This is especially true insofar as humans could build a rival superintelligence that does contest the first AI for those resources.
  • Even if the AI views humans as no competition and no threat, humanity is likely to die incidentally, just by being at ground zero. The AI in this scenario may only want a few solar systems’ worth of resources, but the AI’s efforts still all begin on Earth. The most straightforward way to acquire those solar systems will be to extract the Earth’s resources, rendering it uninhabitable. The AI in this scenario could fully achieve its goals without killing off humanity, but if the AI doesn’t care at all about humanity, then it won’t necessarily bother.

• If a satisfiable AI does want to keep humanity around, this is still unlikely to be good news for humanity, for the reasons discussed in “Won’t AI find us fascinating or historically important?” and “Won’t AIs care at least a little about humans?” (The outlook looks similarly grim if a non-satisfiable AI wants to keep humanity around.)

For more on this topic, see the extended discussions on satisfiability (in the online resources for this chapter, Chapter 5) and making AIs robustly lazy (in the Chapter 3 online resource).