The industry race to achieve smarter-than-human AI is heating up. Against that backdrop, there’s something especially tragic about the idea that humanity may end up destroying itself because some critical subset of the voting public or elected officials thinks that machine superintelligence is an impossible pipe dream. People who think that a machine could never be truly intelligent are liable to be blindsided by what’s coming.

It’s tragic in part because we’ve been here before.

The idea that human engineering can one day do what biology does has been a repeated source of debate and controversy for at least the last three hundred years, and arguably much longer.

In the past, during the heyday of the “vitalists,” it was contentious whether mere inanimate matter could ever become animated at all, in the fashion of the machines we now call “robots.”

If you read an organic chemistry textbook, it will probably mention as a landmark discovery Friedrich Wöhler’s artificial synthesis of urea, a component of urine, in 1828. This was a big deal worthy of mention in textbooks because — for the first time — mere chemistry had duplicated a product of biology, showing that biological and non-biological processes weren’t as unrelated as the vitalists had thought.

It may be hard for readers today to sympathize with the shock felt by earlier scientists on finding out that the products of Life Itself could be duplicated by merely chemical means.

You, the reader, have always lived in a world where biochemistry is chemistry, and there is nothing the tiniest bit miraculous-sounding about hearing that someone has used unliving means to synthesize a byproduct of life. It is perhaps hard to imagine what it would feel like to put such an ordinary and mundane thing as biochemistry into the realm of the sacred. Isn’t synthesizing a biochemical an inherently mundane sort of thing to do? Our scientific forebears must have been fools, one instinctively thinks.

Lord Kelvin, the great 19th-century inventor and pioneer of the field of thermodynamics, seems to somehow have been afflicted by a similar madness: seeing something sacred, holy, and mysterious in aspects of biology that sensible people (people like us, that is, living in sensible times) know to be perfectly mundane science. Quoting Kelvin:

It seemed to me then, and it still seems to me, most probable that the animal body does not act as a thermodynamic engine […] The influence of animal or vegetable life on matter is infinitely beyond the range of any scientific inquiry hitherto entered on. Its power of directing the motions of moving particles, in the demonstrated daily miracle of our human free-will, and in the growth of generation after generation of plants from a single seed, are infinitely different from any possible result of the fortuitous concourse of atoms.

The modern reader might be inclined to look back with scorn on this ancient habit of thought — on these old-time scientists, so deluded as to see mystery in phenomena that surely ought to feel inherently non-mysterious.

Of course chemistry can imitate biochemistry.

Of course DNA copying itself, and directing cells that divide and differentiate, explains in an unremarkable way how generation after generation of trees can come from one acorn.

Of course neurons flashing pulses of chemicals among themselves can compute information and direct your arm to move, and of course a computer can be used to direct a robot arm at least as well as your brain can direct your own limb.

But it wasn’t obvious at the time, to Lord Kelvin. He had not seen an X-ray image of DNA. He had not seen the tiny machines inside us; he had no inkling of the sliding fibers that contract our muscles in response to electrical signals passing down our neurons.

Lord Kelvin had very little understanding of how bodies could possibly work, and in his ignorance, he imagined them to be mystical.

Today, humanity has very little in the way of detailed understanding of how intelligence works. (See Chapter 2 for more on how AI researchers do not understand the AIs they create.) So it’s easy to imagine that intelligence must be mystical.

Ten years ago, some people sagely questioned whether the mechanical motions of automata could ever manage to make art or poetry. Sure, AI could tackle chess. But chess is a cold, logical endeavor, nothing like the creative arts!

Today, of course, the same people instead sagely realize that it would not be hard at all for a computer to just make some pretty pictures; making pretty pictures has always been part of the proper realm of machines. It was obvious to some that computers would be able to produce images that are more attractive to the human eye than human art, but now studies are showing a significant preference for AI art. And of course, there’s surely still an open question as to whether any mere machine will ever generate art with real soul in it, right?

It is by no means certain (the skeptic says), nor even probable, that the vital essence of brain-created art is something that could ever be duplicated by the mere concourse of atoms — or at least, the mere concourse of silicon atoms.

But that’s not how this really works. Human brains are amazing things, but they aren’t magic. Brains are made of parts. These parts can, in principle, be understood, and computers can, in principle, be built to do the same things.

In many cases, we know the underlying biochemistry behind what the brain does. And in all cases, we know the underlying physics of atoms.

In most cases, we don’t know the meaning, the higher-level patterns that allow the brain to do the work it does.^* But the overwhelming lesson of the sweep of centuries through human history is that this state of scientific mystery is temporary.

If I flip a coin and then don’t show you how it landed, your ignorance about the coin is a fact about you, not a fact about the coin. The coin is not fundamentally ineffable. Maybe I even glanced at it before hiding it; maybe I know and you don’t. A blank map does not correspond to a blank territory.

Mystery is a property of questions, not a property of answers. Which is why history is littered with examples where some supremely “mysterious,” “ineffable” phenomenon like the animation of bodies turns out to be continuous with totally mundane aspects of the natural world.^†

The lesson of history so far seems to be that the universe is all, in the end, of a piece. There are no divisions within physics that correspond to the different college buildings where people study different subjects. The international relations department, the physics department, the psychology department, the cellular biology department — at the lowest level, they’re all really talking about the same world, governed by the same underlying laws.

When someone says: “The human brain does this ‘intelligence’ thing. Therefore intelligence is doable in physical principle. Therefore engineers can probably eventually invent some machine that also does intelligence,” they are speaking from the peak of a mountain of similar-sounding guesses that have been vindicated, time and time again, by scientists and engineers across the decades and centuries. Yes, even when it seems vastly counterintuitive; that part is precedented too.

This winning streak is tricky to appreciate, because nobody living today remembers how supremely mysterious phenomena like fire and astronomy and biochemistry and chess-playing felt in centuries past. They are understood today, we grow up knowing that those things are made of mundane parts, and so it feels as if they have always been obviously mundane. It is only the frontier that feels fresh and deeply mysterious.

And so the lesson goes unlearned, and history repeats itself.

Notes