It could help with solving the alignment problem.

The AI alignment problem does not look to us like it is fundamentally unsolvable. It only looks to us that humans are nowhere close to solving it, and that humans are not at the level of intelligence where thinking they have a solution strongly correlates with them actually having a solution.

AI researchers often recognize that the alignment problem looks formidably difficult, and that very little progress has been made on the problem to date. This is why “maybe we can get the AIs to do our alignment homework for us” has had such appeal: When you’re an AI researcher and you don’t feel that you and your colleagues are up to the task of solving a certain problem, the obvious thing to reach for is AI.

But as we discuss in Chapter 11 and in an associated extended discussion, it’s clear even from a layman’s perspective that this idea has many issues: For an AI to figure out how to solve a deep problem that the best human researchers are having a great deal of trouble with, it needs to be smart enough to be dangerous. And since we have very little idea of what we’re doing, we have no source of ground truth that we can use to directly train for narrow alignment capabilities, and no way of checking whether an AI-generated alignment proposal is safe or effective.

The world is allowed to hand us problems that are legitimately out of reach. Nature isn’t a game that only gives humanity “fair” challenges; we can sometimes run into problems that are too hard for even top human scientists to solve, or too hard to solve within the required timeframe.

Is there a more realistic method of passing the whole problem off to some smarter entity? One option would be to make humans smarter in such a way that they might legitimately be able to solve the alignment problem. Humans come “pre-aligned” in a way that AIs don’t; the smartest humans have the same basic prosocial motivations as the rest of us.

In principle, it seems possible for people to be able to tell the difference between what seems like great alchemical enlightenment that will let them transmute lead into gold, and the sort of knowledge that actually corresponds to the ability to transmute lead into gold (using nuclear physics to knock some neutrons off of lead atoms). They should surely feel like different states of knowledge.

But actual human engineers have a lot of trouble telling which zone they’re in. In the actual history of chemistry, humanity’s skill level was such that the alchemists were reliably fooled.

In the real world, scientists get attached to their pet theories and refuse to revise their views until reality hammers them over the head repeatedly with “your theory was wrong” — and sometimes they refuse to change their view even then: Science is sometimes said to advance “one funeral at a time,” because the old guard will never change their views and you’ve just got to wait for the new guard to mature. But this isn’t a fundamental constraint imposed by nature; it’s just an issue of humans as a class being insufficiently savvy, careful, and self-aware.

Usually, it’s okay for humans to be naive in these ways, because usually reality is fairly forgiving of errors, at least in the sense that it doesn’t wipe out all of humanity for the hubris of one alchemist. But that’s not a luxury humanity has when it comes to aligning machine superintelligence.

Humanity often gains its knowledge by struggling, and trying, and failing, and slowly accumulating knowledge. But it doesn’t have to be that way.

Einstein was not only able to figure out general relativity; he was able to figure it out by thinking hard about the problem, even before humanity put satellites in orbit and started seeing discrepancies in their clocks with their own two eyes (as discussed in Chapter 6). He had empirical evidence, but he was able to efficiently pinpoint the right answer in response to the first quiet whispers from the empirical record, rather than needing the truth to come banging at his door.

That pathway is rarer and harder to walk, but that kind of scientific genius does exist — albeit rarely, even among the world’s best and brightest.

Humans augmented one or two steps beyond the level of researchers like Einstein or John von Neumann might begin to accurately figure out their own flaws, and correct for them, in dozens of different ways.

They might notice when they were rationalizing or falling victim to confirmation bias. They might go past the point of ever expecting a clever-sounding idea to work when actually it does not work — to the point where whenever they expect to succeed, they do succeed. They might achieve a level of competence where they still make plenty of mistakes, but they aren’t systematically overconfident (or underconfident) in tricky new domains.

Is human intelligence enhancement really a possibility? It seems so to us, having spoken with a number of biotech researchers who think that there are promising near-term angles of attack. Carefully targeted biotech-focused AI might also help accelerate the work. But from our perspective, it remains very uncertain whether a plan like this would realistically pan out. What we feel more confident in saying is that it’s a highly leveraged option that deserves a lot more investment and exploration than it’s currently getting.

We are not recommending enhancing human intelligence as the only post-AI-shutdown strategy we think humanity should heavily invest in. Rather, this is just one of many examples, and the one we currently think holds the most promise. We strongly recommend that humanity look into multiple possible non-AI paths forward, rather than putting all its eggs in one basket.

Augmented humans don’t pose a major “human alignment” problem.

Augmented humans would have essentially the same brain architecture, emotions, etc., as the rest of us. With AI — even AI trained to sound like us — there’s an enormous gulf of cognitive and motivational difference, and a similarly large comprehensibility gap; with modestly smarter humans, none of that seems particularly likely to be true.^*

Cognitively enhanced researchers wouldn’t need to hold together their own mental integrity while turning into vast superintelligences with minds millions of times larger. They would only need to be raised to the level required to figure out how to build — not grow — artificial superintelligences that would be truly aligned and stable.

There may still be a “human alignment” problem in the weak sense that any effort to coordinate multiple people can run into principal-agent problems and incentive problems. And these problems inherently matter a great deal more with any group tasked with creating superintelligence.

We expect these problems are tractable as long as the humans start out visibly altruistic and charitable, so long as their intelligence is enhanced only slowly, and so long as they work in a well-designed institution with well-designed incentives. But it’s entirely reasonable for people to worry about the potential for a power grab here. Solving these problems wouldn’t necessarily be easy, but it wouldn’t be as fundamentally unfeasible as corporations trying to grow inscrutable superintelligences with entirely incomprehensible minds and inhuman drives.

Creating a crack team of genetically engineered supergeniuses to help navigate the planet safely through the transition to superintelligence is definitely the sort of thing that humanity should do carefully, given the high stakes of such an endeavor. A move like this comes with various practical and ethical issues, but these have to be weighed against the cost of letting superintelligence kill us all, if no other solutions seem similarly promising.

Drastic times can call for drastic measures, but (modest) human intelligence enhancement isn’t even a measure that seems particularly drastic. It seems like a net-positive technology on its own terms, that has at least some chance of helping humanity out in more ways than one.

If you’re interested in helping, let us know.

If you have expertise in the sort of biotechnology that would be useful when it comes to augmenting human intelligence, we encourage you to reach out to us via this form. That goes for funders, researchers, and policymakers alike. We are not experts in the relevant aspects of biotechnology, but we have connections to various figures in the field. At the very least, we can perhaps connect funders and researchers and policymakers to one another, to help grease the gears.

We can work together to stop superintelligence while disagreeing on human enhancement.

If you don’t agree with us about the human augmentation idea, we can still shake hands with you on shutting down frontier AI development.

If we don’t solve that part, everyone is dead. Everyone who doesn’t want to die today has to cooperate to that end. We can wait until after we’re past the threat of immediate death to argue about whether augmenting human intelligence should be illegal or subsidized.

“Make humans smarter than Einstein” is not a plan for how not to die in 2028 or 2032 or whenever there’s the next basic breakthrough in AI algorithms.

It’s not a plan that can run alongside AI development. Even if someone uses medical technology unlocked by dumber-than-human AIs to augment human intelligence well past the Einstein level, those augments probably wouldn’t be able to solve the AI alignment problem and safely design, craft, and build machine superintelligence quickly, under the time pressure of an arms race. The race toward superintelligence still needs to stop.

The idea behind human intelligence augmentation is that it might make it possible at all to solve the alignment problem, if a large number of upgraded researchers also have a significant number of years or decades to work on the problem. The idea is not that they could win a race to build aligned superintelligence six years from now, faster than the rest of the AI industry can build and deploy unaligned superintelligence.

Many people who think “augment human intelligence” is a relatively promising plan — ourselves included — think the first steps still involve shutting down the AI companies.

Those who have other plans about what humanity should do next also generally agree that the first step should be shutting down the AI companies.

Meta AI can’t exist, OpenAI can’t exist, Anthropic can’t exist; they will just kill us. We can agree on this immediate priority, even if we have very different ideas about what to do next.