We noted the example of Jürgen Schmidhuber, a pioneering AI researcher who believed that an AI with preferences for making things as simple as possible would end up loving humans, because humans are such good simplifiers.

In our experience, this is a remarkably common kind of mistake. “Well, the AI will probably end up with aesthetic preferences. And humans make art! So the AI will want to keep us around to make art.”

A recent example comes from xAI, a major AI lab (founded by Elon Musk) whose stated plan for how we all survive is that they’ll make their AI care maximally about “truth” and “curiosity,” and since humans generate truths and inspire curiosity, it’ll all turn out okay. (More on this lab’s plan and other plans the labs’ survival plans, can be found in Chapter 11.)

To really illustrate the problem with this sort of reasoning, it helps to study an example in detail. Let’s take an example that’s a little more neutral than “art,” such as “symmetry.”

Suppose that AI labs used current techniques to grow smarter-than-human AIs that care about symmetry. Would that symmetry preference alone result in care for humans?

You could make the argument, in the fashion of Schmidhuber: Humans are bilaterally symmetrical! How could any AI with a love for symmetry bear to kill anything so symmetrical as us? And you could make other arguments too, like: Humans produce lots of car wheels, which are very symmetric! Why would an AI remove us from the world, when we’re an automatic pre-existing source of symmetrical stuff?

The problem with this reasoning is that it is possible to take the atoms making up a human being, and arrange them in even more symmetrical ways. Or to rearrange the atoms that make up human civilization into factories that produce symmetrical objects even more efficiently. It’s the same mistake the movie The Matrix makes, when it imagines that AIs might keep humans alive in pods as generators of heat and electricity: There are more efficient ways of generating heat and electricity.

For argument’s sake, though, suppose that we imagine that AIs do prize a very specific and unusual kind of symmetry that really does view humans as amazing specimens of symmetry. Even then, why would this preference alone imply that the human beings alive today get to keep living, free and in good health and having fun?

Think like an AI. Even if the AI has to stick with humans, the humans alive today are not the most symmetrical possible human beings. The AI should be able to get even more of its symmetry preference satisfied by repeatedly cloning the most symmetrical living human, or by genetically engineering “improved” humans.

Likewise: Letting those humans just run around is not the cheapest way to keep them alive and symmetrical. Probably they wind up in farms. By storing the humans in a cheap and space-efficient way, the AI can get away with making even more symmetrical humans.

By comparison: Humanity at present has no more efficient way to make eggs than having chickens lay them. As a result, factory farms, whose executives mainly cared about egg count, ended up putting chickens in some incredibly unpleasant conditions, because that was the cheapest way of getting the most eggs.

Similarly, the chickens that existed one thousand years ago weren’t the most efficient possible egg-layers, so farmers bred faster-laying chickens. The chickens of one thousand years ago didn’t grow as much meat as possible, as fast as possible. So now, some modern chickens grow breasts so huge that they cannot walk.

Some humans dislike that we treat chickens this way, and those people bring pressure to bear on factory farms to be less like that — because those humans have additional preferences, beyond a preference for cheap eggs. For that pressure to exist, it is necessary that someone with some power care directly about the wellbeing of chickens at least a little, because taking good care of chickens is not motivated by a preference that is solely about extracting eggs. An AI could, in theory, have other preferences about humans that make it treat us well, but it wouldn’t come from a preference for symmetry (or truth, or simple explanations, or any other preference that’s not actually about us).

Even farmers who have less impersonal relationships with their cattle will prohibit their livestock from mating however they choose. Cattle breeding is serious business, and impinges too much on the future profitability of the farm to let the bulls and cows just go at it.

And even this arrangement will not persist forever. Making beef using cows is very costly in agricultural land, and a number of current startups are trying to synthesize beef more directly.

Synthetic meat is not an easy engineering problem at our tech level. Humanity is only just starting to catch up with some of what natural selection does in the way of organic chemistry. But if humanity were better at rearranging atoms, there would be many fewer cows — cows are not maximally fun to keep around, if you don’t need them for milk and meat.

So things aren’t looking good for the assumption we began with, for argument’s sake — that an AI with alien preferences would keep humans around forever, in the name of “symmetry.” Even in the unlikely case where the AI has a very strange notion of “symmetry” that ranks humans very highly, it’s a lot harder to find a notion of symmetry that considers humans optimal. Either way, things aren’t looking good for humanity.

Realistically, a symmetry-loving superintelligence would not keep humans alive; if it kept us alive, there is no real chance that it would keep us healthy and happy and free. We’ve stacked on far too many nice-sounding coincidences, at that point. If the AI specifically cared about our welfare and wanted us to be happy for that reason, then that’s one thing. But to imagine that far simpler, easier goals suffice seems like a fantasy.

All of these arguments apply with equal force to “just make an AI that values truth” or “just make an AI that values beauty.” It’s just that those cases make it easier to get lost in the fantasy, because words like “truth” and “beauty” sound intuitively nicer than “symmetry.”

If something sounds nice as a slogan (“make the AI value truth above all else!”), then the temptation is to imagine that it would have nice consequences as a policy. The temptation is to imagine that the virtues all go together, so that endorsing one good thing means that the other good things will come along for the ride. But nature, and machine learning, are less kind than that.

Instead of leaving the idea pleasantly vague, consider any concrete metric the superintelligence might be optimizing in seeking “truth.” Then observe that humans won’t be the maximum of that preference for learning truths. They won’t be anywhere close.

Even in the unlikely event that the AI gravitated specifically to the kinds of truths that humans tend to express (rather than, e.g., random arithmetic equations), the best way to get more of those truths wouldn’t be by keeping humans around and using them to generate human-style conversations.

And either way, the actual present-day human population — the actual human beings alive today, your friends, your family, you — wouldn’t be among the cheapest-to-feed, tastiest-to-milk domesticated “truth” producers.

Happy, healthy, free people leading flourishing lives are not the most efficient solution to almost any problem. For an AI to keep us alive and well, it has to care about us at least a little.