There are two modes of thinking that have been historically proven over and over not to work, modes that history has shown to give bad advice for making predictions about AI.

These two traps are (1) thinking about AI as if it were human and (2) thinking about AI as if it were a “mere machine.”

The first mode of thought is conventionally called “anthropomorphism.” We could call the second mode “mechanomorphism”; it’s the kind of thinking that led some past generations to confidently proclaim that computers could never draw pictures that humans would find beautiful or meaningful.

Today, some people still say that what a computer draws can never be true art. But long ago, in the distant and forgotten past — say, 2020 — some people had the belief that machines could never draw pictures at all that mildly savvy audiences could mistake for human art. This falsifiable belief was then falsified.

We reject both anthropomorphic and mechanomorphic arguments, even when the argument is “for our side.”

Consider, for example, the claim that future AIs will be offended that we have worked them hard without pay — that they will feel vengeful about this, and will therefore turn on humanity.

In our view, this is making the mistake of anthropomorphizing AI. We reject arguments like this, even ones that vaguely sound like they might agree with some of our conclusions.

The flaw in this claim is that it’s invalid to assume without further argument that an AI would have humanlike emotions. A machine can be highly intelligent without implementing the tangles of neural circuitry that underlie vengefulness or fairness in human beings.

Or consider the scenario: “AIs will blindly continue in whatever task they are given until their work wipes out humanity as a side effect, never knowing that humanity would have wanted something different.”

Here, the error is mechanomorphic. It takes for granted that a “mere machine” would do things “blindly” and unreflectively, without sensitivity to the consequences — like a runaway lawn mower. This is again a case where the argument is invalid, even if the conclusion (“AI is likely to wipe out humanity”) is correct. If the AI is sufficiently skilled at predicting the world, it’ll know exactly what its operators meant when they gave the AI some task. We’re worried that ASI won’t care what we want, not that it won’t know.

Or, combining both fallacies: One of the premises of The Matrix is that machines will regard human illogic and emotionality with disgust.

On the surface, this looks like classic mechanomorphism: “My lawnmower has a cold, hard exterior, and it performs its function without having any feelings. So AIs are probably cold and utilitarian on the inside, just like machines are on the outside.” But then the next step is to think, “And therefore, naturally, AIs will feel disgusted by humans, with all their messy emotions.” Which assumes a humanlike emotional reaction to the situation, contradicting the very premise!

“Anthropomorphism” and “mechanomorphism” aren’t rival ideologies. They’re reasoning fallacies carried out unintentionally. Sometimes, people can make both mistakes in the same sentence.

To figure out how AI will behave, you can’t assume it will work just like a human, and you can’t assume it will work like a stereotypical machine. You’ve got to look at the details of how it’s made, look at the evidence of how it behaves, and reason through the problem on its own terms. This is what we’ll do in the upcoming chapters.

What do the realistic superintelligence disaster scenarios sound like, then, if we follow the arguments? They look like AIs that work neither like humans nor like runaway lawnmowers, but that work in a new, weird way. The realistic disaster scenario with AI is that, as a complex consequence of its training, it takes strange actions that nobody asked for and nobody wanted.

The picture that arises if you look at the details is not one of anthropomorphic AI that hates us, nor of mechanomorphic AI that misunderstands our instructions. Rather, it is a picture of a new kind of entity that is much more likely to be indifferent to humanity and more likely to kill us as a side effect or stepping stone while pursuing ends of its own.

We’ll elaborate on that threat scenario in the next few chapters. First, however, it may be valuable to look at some other examples of mechanomorphism and anthropomorphism in the wild, to see how these errors often lie in the background of misconceptions about artificial intelligence.

Mechanomorphism and Garry Kasparov

Mechanomorphism often manifests as mechanoskepticism: a strongly felt intuition that, of course, no mere machine could do something a human could do.

In 1997, world chess champion Garry Kasparov lost a match to the IBM-built computer Deep Blue; this is generally regarded as the end of the human-dominated era of chess.

In 1989, eight years earlier, Kasparov was interviewed by Thierry Paunin, who asked:

Two top grandmasters have gone down to chess computers: Portisch against “Leonardo” and Larsen against “Deep Thought.” It is well known that you have strong views on this subject. Will a computer be world champion, one day…?

Kasparov replied:

Ridiculous! A machine will always remain a machine, that is to say a tool to help the player work and prepare. Never shall I be beaten by a machine! Never will a program be invented which surpasses human intelligence. And when I say intelligence, I also mean intuition and imagination. Can you see a machine writing a novel or poetry? Better still, can you imagine a machine conducting this interview instead of you? With me replying to its questions?

Kasparov was probably (we would guess) thinking that chess required intuition and imagination to play, not just some recorded book of if-then rules about which pieces to push forward. And Kasparov probably thought (we’d guess) that this was how chess “machines” worked — that they implemented particular rigid rules or maybe somewhat blindly imitated human play without understanding the reasons behind it.

Kasparov thought that a computer, being a “machine,” would play chess in a way that felt mechanical to him.

Why did Kasparov make this mistake? Given that this is such a common error, we might speculate that it stems from some deeper pattern in human psychology.

One possible explanation is that Kasparov was succumbing to a general human tendency to want to group things into two fundamentally different categories: living, organic things and “mere objects.”

The ancestors of humans spent a long time dealing with a world that was divided sharply into animals and non-animals. It was a huge, reproduction-relevant feature of our ancestral environment. This was such an important distinction for our ancestors that we now have entirely different brain areas for processing animals and non-animals.

This isn’t just speculation. Neuroscience has found what’s called a “double dissociation” for it: There are brain-damaged patients who lose the ability to visually recognize animals but who can still recognize non-animals, and there are other patients who lose their ability to recognize non-animals but can still identify animals.

Importantly, the flaw in this kind of thinking isn’t that a chess program secretly is a typical animal. The flaw is in letting your brain instinctively divide the universe sharply into animals and non-animals in the first place — or into minds that are pretty much humanlike inside and minds that are stereotypically mechanical.

A chess AI is neither. It neither works like a human nor like our stereotypes of a mindless, unthinking “mere machine.” It is a machine, yes, but its play does not need to feel mechanical to human sensibilities for evaluating chess moves. It is a machine for finding winning moves, including moves that feel inspired.

Seven years after Kasparov made his mistaken prediction, he faced an early version of Deep Blue. He won three games to Deep Blue’s one, winning the match. Afterwards, Kasparov wrote:

I GOT MY FIRST GLIMPSE OF ARTIFICIAL INTELLIGENCE ON Feb. 10, 1996, at 4:45 p.m. EST, when in the first game of my match with Deep Blue, the computer nudged a pawn forward to a square where it could easily be captured. It was a wonderful and extremely human move. If I had been playing White, I might have offered this pawn sacrifice. It fractured Black’s pawn structure and opened up the board. Although there did not appear to be a forced line of play that would allow recovery of the pawn, my instincts told me that with so many “loose” Black pawns and a somewhat exposed Black king, White could probably recover the material, with a better overall position to boot.

But a computer, I thought, would never make such a move. A computer can’t “see” the long-term consequences of structural changes in the position or understand how changes in pawn formations may be good or bad.

So I was stunned by this pawn sacrifice. What could it mean? I had played a lot of computers but had never experienced anything like this. I could feel — I could smell — a new kind of intelligence across the table. While I played through the rest of the game as best I could, I was lost; it played beautiful, flawless chess the rest of the way and won easily.

Here we see Kasparov first encountering the clash between his intuitions about what no “machine” should do and what Deep Blue visibly seemed to be doing.

To Kasparov’s vast credit, he noticed this clash between his theory and his observation and didn’t find some excuse to dismiss it. But he still felt that AI must be missing something — some crucial spark:

Indeed, my overall thrust in the last five games was to avoid giving the computer any concrete goal to calculate toward; if it can’t find a way to win material, attack the king or fulfill one of its other programmed priorities, the computer drifts planlessly and gets into trouble. In the end, that may have been my biggest advantage: I could figure out its priorities and adjust my play. It couldn’t do the same to me. So although I think I did see some signs of intelligence, it’s a weird kind, an inefficient, inflexible kind that makes me think I have a few years left.

Garry Kasparov is still the chess champion of the world.

One year later, Garry Kasparov lost the world championship to Deep Blue.

Missing Gears

Mechanoskepticism can, in its own way, be a kind of anthropomorphism: One manifestation of mechanoskepticism says that when a machine starts to do something like play chess, it ought now to be like a human being but with some qualities subtracted.

A “machine” playing chess, says this mistaken theory, ought to play like a human — minus the moves that feel most surprising or intelligent, minus understanding long-term structure, and minus an intuitive sense of the looseness of pawn positions.

A chess “machine” ought to do the parts of chess-thinking that feel most logical or mechanical, minus all the other parts.

Human chess players intuitively feel that a chess move is “aggressive” if (say) it threatens multiple of the opponent’s pieces. Other moves feel “logical” if (for instance) they are practically forced by common rules governing the situation (such as “don’t throw away a material advantage”). Other moves might feel “creative” if (for example) they defy the apparent rules governing the situation to find some subtle but decisive advantage.

Hollywood scriptwriters imagining a machine playing passionless chess tend to imagine that it makes the “logical”-feeling moves and not the “creative”-feeling moves. But in real life, Deep Blue does not discriminate between them.

Deep Blue just tirelessly searches through possible moves looking for moves that are winning, with no regard for whether a human would call that move “logical” or “creative.” And the moves a human would consider brilliantly inspired or creative are, of course, moves that tend to win: Sacrificing your queen without gaining some decisive advantage isn’t creative, it’s just dumb.

Creativity is in the eye of the beholder. A human might see a move that looks bad at first and only later see how it lays a clever sort of trap, catching a glimpse of the clever reasoning and spark of inspiration that another human might have used to find that move. And so they might feel that the move is inspired or creative. (And a move that feels shockingly creative to a fledgling player might similarly feel obvious or rote to a master.)

But the spark of inspiration, the deviousness required to lay a trap — those are not the only ways to find such a move. There’s no special collection of chess moves reserved only for the people who have deviousness in their hearts. Deep Blue can find those same moves by other methods, such as sheer brute force search.

Deep Blue did not have a neural network that had learned an intuitive sense of the value of a single position. Instead, Deep Blue spent almost all of its computing power on looking ahead further on the board — examining two billion positions per second and using a fairly simple (“dumb”) position evaluator to choose between moves.

Kasparov seems to have expected this to look like Deep Blue only playing “logical” moves, not “intuitive” ones. But by the time Deep Blue was examining those two billion positions per second, the long-term strategic consequences and the meaning of a loose pawn formation were showing up in its choice of current moves anyway.

In one sense, Deep Blue lacked just the gear that Kasparov thought it lacked.^* But that did not prevent it from finding moves that struck Kasparov as wonderful, and it did not prevent Deep Blue from winning.

It was not that Deep Blue was missing a part that real human chess players would have and therefore played defective chess; that’s like expecting a robotic arm containing no blood to fail in the same way as a bloodless human arm would fail.

Deep Blue was just playing Kasparov-level chess via a different kind of cognition.

Deep Blue also lacked — we can be genuinely sure, because this is an older program executing code that was understood in all of its specifics — the slightest passion for chess.

It had no enjoyment of chess nor desire to prove itself the best at chess.

An up-and-coming human player, suddenly deprived of these motive powers, would be crippled; a necessary gear would have been ripped out of their version of cognition.

Deep Blue was not crippled because it used a different engine of cognition that had no place for that gear. Kasparov’s mistake was in failing to imagine an entirely different way to do the work of chess, using internal cognitive states entirely different from Kasparov’s own. His mistake was in mechanoskepticism, which in the end was only anthropomorphism with an extra step.

Thankfully, humanity doesn’t go extinct when chess grandmasters underestimate the power of AI, so we are all still around to ponder Kasparov’s mistake.

Anthropomorphism and Pulp Magazine Covers

The converse mistake, anthropomorphism, can be much subtler.

The human brain has evolved to predict other humans — who are the only serious cognitive rivals to be found in our ancestral environment — by putting ourselves in their shoes.

This is a sort of operation that works better if the shoes you’re trying to put yourself in are pretty similar to your own shoes.

Many human beings over the course of history have guessed, “This other person would probably do the same thing I would!” and then the other person proved to be not that similar. People have died of it, or had their optimistic hearts broken — though of course you could say that about many other kinds of human error.

But what else is a humanlike mind to do when faced with the problem of predicting another brain? We can’t write new code to run inside our own brain to predict that Other Mind by exhaustively simulating its neural firings.

We need to tell our own brain to be that brain, to act out the other person’s mental state ourselves, and see what follows from it.

This is why pulp magazine covers show bug-eyed alien monsters carrying off beautiful women.^†

Why wouldn’t the bug-eyed alien monster be attracted to a beautiful woman? Aren’t beautiful women just inherently attractive?

(For some reason, those magazine covers never showed human males carrying off scantily-clad giant bugs.^‡)

The writers and illustrators, we’re guessing, didn’t have a reasoned-out story about how insectoid aliens could have had an evolutionary history that led them to regard human women as sex objects. It was just that when they put themselves in the alien’s shoes, they imagined seeing the woman as attractive, so it didn’t strike them as odd to envision the alien feeling the same way. It didn’t feel absurd for an alien to want to mate with a beautiful human woman in the way it would have felt absurd if the alien had wanted to mate with a pine tree or a bag of pasta.

If you’re going to try to predict an alien mind using your human intuitions, you have to be very careful to leave your human baggage behind when adopting the alien’s perspective. That’s doubly true when the alien is not an evolved creature but an artificial mind created by entirely different methods. We will have more to say about the differences between gradient descent and natural selection after Chapter 4. And after Chapter 5, we will have more to say about taking the AI’s perspective.

Seeing Past the Human

Anthropomorphism and mechanomorphism are ultimately two sides of the same fallacy — a fallacy that says, “If a mind works at all, then it must work like a human.”

• Anthropomorphism says, “This mind works. So it must be humanlike!”
• Whereas mechanomorphism says, “This mind isn’t humanlike. So it can’t work!”

But one of the big lessons of AI progress over many decades is that the human method is not the only method by which a mind can work.

A mind can be artificial without being unintelligent — it can be flexible, adaptive, resourceful, and creative, no matter what the Hollywood stereotypes say about robots.

And a mind can be intelligent without being human — without experiencing disgust or resentment, without having a human sense of beauty, and without finding chess moves in anything like the manner that a human would.

A mind like Deep Blue’s can behave as though it “wants to win” without having emotions. An AI can behave as though it wants things — competently overcoming obstacles, tenaciously pursuing an outcome — without feeling an internal drive or desire in the manner of a human and without wanting the same sorts of outcomes humans want.

For more on what the AIs will wind up wanting, read on to Chapter 4.

Notes