Kindness looks contingent on the particulars of our biology and ancestry.

Kindness does not look like the sort of property that every mind ends up with, for a variety of reasons. Here’s four, which we cover in more depth in the extended discussions:

1. Curiosity Isn’t Convergent: Traits like curiosity and boredom help humans solve specific mental challenges, like the challenge of understanding the environment. But there are other ways to solve those challenges, and AIs are likely to solve them in different ways. Submarines move through the water just fine, but they don’t quite “swim.” Many other traits, including kindness, can be understood analogously.
2. Human Values Are Contingent: Humans evolved traits like kindness and empathy due to the details of our biology and ancestry. It was plausibly important, for example, that humans evolved in tribal groups where we had limited ability to deceive others and limited ability to track how related different tribe members were.
3. Deep Differences Between AIs and Evolved Species: Evolution and gradient descent work very differently, and both processes are very unpredictable. Even if you re-ran evolution on primates, it’s not clear that you would reliably get traits like kindness and true friendship a second time.
4. Reflection and Self-Modification Make It All Harder: Even in the unlikely event that AIs start off with a measure of kindness, they might not preserve their kindness as they become smarter and change in various ways.