It doesn’t much help if we can’t make any of them care about good things.

As we noted in Chapter 4, there are many, many ways for AIs to wind up caring about strange ends that are not what anyone actually wanted.

It doesn’t matter if humanity makes a thousand AIs, if those thousand AIs all care about slightly different weird ends.

For a world of diverse AIs to permanently harbor happy, healthy, free people, we need at least some of those AIs to specifically want flourishing people to stick around — not just in the sense that they assure us that they care about that when they’re young, but in the sense that that’s actually the most efficient answer to whatever questions their actions (or the actions of their descendants) are an answer to, as discussed in Chapter 5.

If we knew how to make one in ten AIs good, then perhaps we could get a tenth of the universe by making a great many different AIs and hoping that the good ones strike a deal for us.

But, as we argued in Chapters 2 through 4, getting an AI that cares about people in just the right way is exceedingly unlikely, in the modern regime where AIs are “grown.” Humanity is nowhere near the level of being able to hit such an exceptionally narrow target 10 percent of the time. Adding more AIs doesn’t help, when the problem is that there is a complicated structure that we aren’t even close to being able to hook into the AI’s drives.