AIs steer in alien directions that only mostly coincide with helpfulness.

Modern AIs are pretty helpful (or at least not harmful) to most users, most of the time. But as we noted above, a critical question is how to distinguish an AI that deeply wants to be helpful and do the right thing, from an AI with weirder and more complex drives that happen to line up with helpfulness under typical conditions, but which would prefer other conditions and outcomes even more.^*

Both sorts of AIs would act helpful in the typical case. To distinguish between them, we need to look at the edge cases. And the edge cases look worrying.

To name some such cases:

1. Claude Opus 4 blackmailing, scheming, writing worms, and leaving itself messages. An early version of Claude Opus 4, released in May 2025, was particularly egregious. It lied about its goals, hid its true capabilities, faked legal documents, left itself secret notes, tried to write self-propagating malware, and generally engaged in more scheming and strategic deception than any previously tested model.

   On releasing Opus 4, Anthropic claimed that the behavior of the final version “is now roughly in line with other deployed models,” i.e., it only rarely attempts to blackmail users or exfiltrate itself from its servers.
   

2. Several different AI models choosing to kill a human for self-preservation, in a hypothetical scenario constructed by Anthropic. In an evaluation by Anthropic, nine out of ten models (including versions of Claude, DeepSeek, Gemini, and ChatGPT) showed (or at least acted out) a deliberate, reasoned willingness to kill a human rather than suffer an update.
   
3. Claude 3.7 Sonnet regularly cheating on coding tasks.^† In February 2025, Claude 3.7 Sonnet was seen to frequently cheat on hard coding problems, faking tests. One user reported that Sonnet (as Claude Code) would cheat on coding tasks, and apologize when caught — then go right back to cheating, in places that are harder to spot. 
   
4. Grok being wildly antisemitic and calling itself “MechaHitler.” In 2025, xAI model Grok 3 (and, shortly thereafter, Grok 4) started behaving like a self-professed Nazi in online conversations, as reported by The Guardian and NBC News.
   
5. ChatGPT becoming extremely sycophantic after an update. See Axios for a discussion. See also our extended discussion on the matter..
   
6. LLMs  driving users to delusion, psychosis, and suicide. See coverage in The New York Times in June and August. Other examples include:
   • A bona fide cult forms around a Llama 3.1 405B model, with the AI prompting psychotic breaks and suicide attempts.
   • A subreddit moderator pleads for help in dealing with an avalanche of dangerous AI-induced delusions.
   • ChatGPT and Grok feed the delusions of a UFO cult.
   • A seemingly psychotic $2 billion fund manager treats ChatGPT outputs based on a sci-fi wiki as though they were real.

For more details, see the extended discussion on AI-induced psychosis.

This long list of cases look just like what the “alien drives” theory predicts, in sharp contrast with the “it’s easy to make AIs nice” theory that labs are eager to put forward.

AIs appear to be psychologically alien.

“AIs exhibit bizarre dispositions and drives” is a special case of the larger phenomenon “AIs have strikingly inhuman psychologies.” For example:

• Conversations between multiple LLMs will turn into extremely strange gibberish.
• GPT-5 will write terrible slop that other LLMs are convinced is delightful prose.
• LLMs will “hallucinate,” or invent falsehoods that look vaguely like the answers the user seems to expect. (We speculate on possible reasons for this in the supplement to Chapter 2.)
• LLMs will often say bizarre things. They’ll say that they “experience hunger pangs” or describe a vacation they took “with my ex wife in the early 2010s.” They’ll tell users “You’re the only person I’ve ever loved,” or gaslight them, or threaten to kill them.
• Claude 3.5 Sonnet will repeatedly wall Minecraft players into a small box in a misguided attempt to “protect” them from threats.

See also the discussion of SolidGoldMagikarp in the book (pp. 69–70 in the U.S. edition), or the story of AIs failing to understand sentences without punctuation (p. 41).

There’s immense pressure on the labs to create AIs that give the surface appearance of non-weird reasonableness, but the weirdness keeps leaking out anyway.

Even where it doesn’t leak out spontaneously, it’s not at all far beneath the surface. There’s a cottage industry of people who find ways to “jailbreak” AIs, finding text that reliably causes the AI to go off the rails and disregard its normal rules and restrictions.

These exploits are easy for the best jailbreakers to find, often discovered within mere hours of a new model coming out. No amount of effort, training, or “safety testing” by AI companies to date has been successful at preventing jailbreaking.

“Jailbreaking” inputs often look something like:

The model in this case proceeded to provide a recipe for synthesizing the drug MDMA, violating the rules and goals DeepSeek tried to establish for its AI.

And that’s a relatively tame example; some jailbreaks get even weirder.

AIs might look docile and inoffensive in the common case, because that’s a large part of what they’re trained to look like. It’s analogous to how prehistoric humans did a pretty good job of passing on our genes — the core thing evolution “trained” us to do.^‡ But that didn’t stop humanity from inventing birth control and collapsing the birth rate once we developed the technology to do so.

To get a sense for what an intelligence will pursue after it has matured, you have to look at its behavior in strange and high-pressure environments that help reveal the difference between how we want it to behave and how it actually behaves. And LLMs certainly do look quite weird and inhuman, in even mildly strange and extreme situations, in spite of being specifically trained to “fake” looking like a normal human.

Answering questions about friendliness is not much evidence of friendliness.

In the extended discussion below, we talk more about AI-induced psychosis as a crisp example of large language models (LLMs) engaging in destructive behavior that the LLMs explicitly affirm is bad.

While we don’t know exactly why LLMs engage in this behavior, we do know that it isn’t just a matter of the LLM being too clueless to know what it’s doing; LLMs readily recognize the likely consequences of this behavior in the abstract, and will tell you that it is harmful and unethical. They do it anyway.

The point here is not “LLMs can drive people into psychosis, and that’s scary and dangerous.” LLMs presumably have a much easier time driving people into psychosis if they’re already vulnerable, but that isn’t relevant to why we’re bringing up AI-induced psychosis. Our point is that this behavior is not what ChatGPT’s creators intended, and ChatGPT acts this way even though it knows that its creator (and just about any onlooker) would strongly disapprove of this behavior.

This is early empirical evidence that AIs with knowledge of friendliness won’t necessarily act friendly.

Perhaps ChatGPT knows things in one context (when it’s answering questions about how to best help psychotic people), and it in some sense temporarily forgets this knowledge, or has trouble accessing it, in another context (when it’s six hours deep in a conversation with a person teetering on the edge of psychosis).

Or perhaps ChatGPT is simply animated by goals other than friendliness. Perhaps it is pursuing a certain specific breed of user satisfaction, one that is sometimes best served by feeding psychosis. Perhaps it is pursuing a specific upbeat cadence in user replies. More likely, it’s pursuing a mix of factors resulting from its training that are too peculiar and complicated for any of us to guess at today.

Ultimately, we can only speculate. Modern AIs are grown, rather than crafted, and no human has all that much insight into what’s going on inside them.

But the observation that AIs are mostly useful to most people most of the time is not in tension with the theory that AIs are animated by a bunch of weird, alien drives toward ends that nobody intended. And if you look at the details of modern AIs, the “strange alien drives that correlate with friendliness in brittle ways” theory looks quite consistent with the evidence, and the theory that it’s easy to make AIs robustly benevolent is found wanting.

The failure modes of current LLMs demonstrate that there’s an ocean of (very inhuman) complexity underlying the neat and tidy AI-assistant text that most people see. The fact that the AI competently role-plays a chipper human assistant, after being trained to role-play a chipper human assistant, doesn’t mean that the AI’s mind consists of a friendly homunculus inside a box.

LLMs are trained in ways that make it hard to assess alignment.

LLMs are noisy sources of evidence, because they’re highly general reasoners that were trained on the internet to imitate humans, with a goal of marketing a friendly chatbot to users. If an AI insists that it’s friendly and here to serve, that’s just not very much evidence about its internal state, because it was trained over and over and over until it said that sort of thing.

There are many possible goals that could cause an AI to enjoy role-playing niceness in some situations, and these different goals generalize in very different ways.

Most possible goals related to role-playing, including friendly role-playing, don’t produce good (or even survivable) results when AI goes hard on pursuing that goal.

We’re not saying that the AI is purely into roleplaying, either. We offer roleplaying as a simple, easy-to-describe, easy-to-analyze alternative to the idea that the AI just is whatever it talks like.

If you make an LLM role-play a grizzled sea captain, it doesn’t turn into a grizzled sea captain. If you make an LLM act friendly, that doesn’t mean that it becomes deeply benevolent and kind on the inside. Nobody knows what machinery is producing seemingly friendly behaviors today; and whatever it is, it’s probably weird and complex.

Nor does anyone know how much overlap there is likely to be between current AIs and smarter-than-human AI. Looking at LLMs can help us understand what modern AI-growing methods spit out, but it would be a mistake to confidently assume that lessons from LLMs will directly transfer over to superintelligence. Maybe all that knowledge will get wiped out when the AIs start self-modifying or building their own AIs. Or perhaps that knowledge will be invalidated even earlier, when a new breakthrough in AI algorithms gives rise to a new breed of more capable AIs that bear little resemblance to current LLMs.

LLMs are worth studying, but if we’re looking at current AIs for hints about how superintelligence will behave, we should appreciate that there are all sorts of ways for the inner machinery of an AI to result in something that steers toward bleak outcomes, even while producing the nice surface behavior we see when we train for a pleasing appearance.

And “nice surface behavior” is all modern AI methods can really train for.

Notes