To a powerful AI, wouldn’t preserving humans be a negligible expense?
There are many negligible expenses, and it would need a reason to pay ours.
Keeping a pile of forty-one stones in your house would be a negligible expense, but you’re almost surely not going out of your way to pay that expense.*
That something would be cheap does not mean it will get done. The AI would still need to care at least a little, and it probably won’t.
One might ask: But Earth intercepts around 0.0000045 percent of the light emitted by the Sun — one part in 2.2 billion. Do all the people worried about AI just fail to understand how large the Solar System really is? Why would AIs need our planet when there’s so much mass and energy lying around?
One answer is that the AI will start out on Earth, which has vast oceans waiting to be heated up and boiled off as coolant for computation. Earth also has matter that could be turned into probes and sent to other stars. Refusing to use up the Earth costs time, and time is important (as galaxies slip away, forever beyond reach).
Even if the AI can easily make it into space and start building large-scale machines without destroying Earth in the process, it is unlikely to ignore the Sun.
One of the more standard theories about how an advanced civilization might grow involves the civilization building a Dyson sphere to capture more of the Sun’s light. Other proposals involve harvesting even more energy by “lifting” matter out of the star to fuse in power plants that capture almost all of the energy released by fusion (rather than letting most of it go to waste in the center of a star).
None of these proposals by default leaves much sunlight falling upon Earth to keep the plants growing and the climate stable. The AI would have to go out of its way to leave us that.
It might still seem like human energy demands are negligible. A human needs around 100 watts of power to live, which is chump change for the sort of entity that can harvest stars. Would a superintelligence not spare even the 800 gigawatts it would take to keep 8 billion humans alive?
We answer, in the end: not unless it cares for that outcome or its consequences more than every other thing it could achieve with 800 gigawatts.
The vast majority of humans do not spare the relatively negligible amounts of sugar it would take to keep the nearest anthill in caloric surplus. Keeping humanity happy would be a negligible expense for an AI that wanted this, but first the AI would need to have that preference. The mere fact that we want it doesn’t mean that the AI will care.†
* Or at least you wouldn’t’ve, before we gave you reason to do it just to spite us.
† Additionally, if the AI did have preferences that involved humans in some way, this probably wouldn’t turn out well for us.
Notes
[1] go out of its way: Stationary gaps in the coverage of the solar cells of a Dyson swarm — or rather, gaps that follow Earth around in orbit — are physically possible, but they wouldn’t be easy to set up (because the orbital velocity of a Dyson swarm between the Earth and the Sun would have to be higher than the orbital velocity of the Earth, in order for the solar cells to stay in orbit while closer to the Sun). And the infrared radiation emitted by the solar panels would cook Earth if it weren’t carefully aimed, and so on. Preserving the Earth is not free, to a superintelligence running large-scale projects in the solar system and beyond. It’s probably possible, but it would require effort.