The human preference for ecological preservation looks like another weird contingent drive.

One hope we’ve heard is that AIs might keep humans around in much the way humans try to preserve nature. Conservationists fight to keep species from going extinct. Being smarter and more capable, AIs should have an easy time protecting humans — that is, assuming AIs want to keep humans around.

We expect this to fail primarily because we expect the AI to wind up with its own strange, complicated desires, rather than with recognizably humanlike desires. For more on this point, refer back to Chapter 4 (and some of the associated extended discussions). For some early empirical evidence on this point, see the discussion on AI psychosis.

Secondarily, even in the unlikely case that an AI somehow ends up with a humanlike desire for “preserving” the world it came into, we don’t think this would turn out very well for us. We think that this sort of reasoning by analogy — “humans preserve the environment, so maybe AIs will preserve us!” — is a kind of wishful thinking.^*

Suppose that, somehow, an AI wound up with a humanlike drive to protect its natural environment. To figure out what would follow, we can start by looking at the actual human drive to protect nature.

Unfortunately, this drive looks spotty at best. Set aside the fact that, when humans have to choose between ecological preservation and some other goal, ecological preservation often loses out. Perhaps that’s just an artifact of humanity’s technological limitations. Perhaps if we had wondrous future technology, we could both have our cake and eat it too.

No, the “spottiness” in our preservation drive that’s relevant to the situation at hand is that, when it comes to ecological preservation, we prefer to preserve the parts of the ecology that seem most interesting or beautiful or otherwise valuable to us, according to all our other drives.

People get up in arms about protecting cute pandas, while unappealing species like the giant earwig and the gastric-brooding frog languish in obscurity until they die out. There are even some species we might prefer to eliminate, like malaria-carrying mosquitoes, which kill half a million children every year.

Most humans don’t have a pure conservationist drive. We have a conservationist drive that’s colored by all our other values.

To drive the point home, consider jewel wasps, screwworm flies, botflies and other similar parasites, which lay their eggs inside living prey; the larvae eat their way out of the host, causing extreme pain in the process. Would the world truly be a better place, according to most people’s values, if we preserved this “natural wonder” exactly as it is? In the limit of technology, could we not at least genetically engineer these parasites to provide a bit of anesthesia here and there? Would it truly be better not to tweak these insects to lay their eggs in plants instead?

Nature is, when one looks beyond the bits that are emphasized to children, full of horrors. It does not seem obvious that, if humans get to have a good future, our descendants would decide to let all these horrors continue. There are already humans who have declared their concern for the welfare of wild animals.

Our preference for preservation is not pure, not simple, not straightforward. It contains internal conflicts and tensions tied to all of our other values and drives.

We don’t know how humanity’s preservation instincts would play out at the limits of technological maturity. The point is: Even if an AI wound up with some drive for ecological preservation, that doesn’t mean humanity gets to have a happy ending. Because any preservation drive that makes it into the AI is also liable to be impure, complex, and jumbled up with all of its other values and drives.

Perhaps, just as according to humanity’s preferences some animal habits are abhorrent, according to the AI’s preferences, some human psychological states would be abhorrent. Just as we’d tweak the screwworm flies so that they stop chewing an agonizing tunnel through living flesh, perhaps the AIs would make a new breed of humans that have music or loneliness edited out of them. Or perhaps AIs would make other, more complicated modifications to humanity, according to complex preferences that we simply cannot predict.

To make an AI that actually lets people lead flourishing lives, we’d probably need to make one that actually cares about that in particular. We’d have to figure out how to make AIs care about us at least a little, and that doesn’t come free.