To provide a challenge to Sable.

If Galvanic, the makers of Sable, stumbled their way into developing a superintelligence without taking any precautions to keep it controlled (such as having AI supervisors and honeypots), readers might feel as though the AI succeeded only because we were being cynical about AI companies.

By depicting Galvanic as being on the more paranoid end of the spectrum (while still trying to keep our depiction of Galvanic realistic), we have more opportunity to demonstrate how an intelligent agent can slip through a web of constraints.

We happen to believe that the most reckless AI companies would be markedly more reckless than Galvanic, for reasons we’ll get into in Chapter 11. And it’s the most reckless companies that matter here, not the most responsible. If three responsible companies avoid building a machine superintelligence because it would be too dangerous, but a fourth irresponsible company rushes ahead, then superintelligence gets built in that fourth lab.

Today, corporate executives at the labs argue, “Better me than them!” Today, executives only exercise as much caution as they can manage without slowing down at all, which we would guess results in less caution than Galvanic is depicted as taking with Sable.

Notes