This seems like a positive update, albeit a minor one.

The relevant experimental results are in a paper called Emergent Misalignment (Betley et al. 2025). Roughly speaking, the paper shows that LLMs tuned to do one bad activity — namely, writing code with flaws in it — also declared themselves to be Nazis and exhibited other bad behavior.

This is a good sign about how it might be possible to tune LLMs to act well on one dimension, and get LLMs that behave well on many different dimensions. We see this as evidence that relatively weak AIs may be more useful than we would have expected, in the regime before we start getting to dangerous capability levels.

Unfortunately, we don’t think that this positive result matters much when it comes to superintelligence, for two reasons.

First, we highly doubt that this “goodness” direction inside the AI is the real deal. If a superintelligence went hard on steering the world in whatever direction that vector points, we doubt the result would be good.

Human value is complicated, and there are many things that correlate with “real goodness” while sometimes diverging in big ways. For instance, perhaps the vector points in a direction that puts too much weight on respecting the social consensus, and too little on discovering socially uncomfortable truths (as suggested by how AIs have trouble making tradeoffs that humans consider obvious). There’s little reason to expect the “goodness” vector to point robustly at goodness, and there are strong empirical and theoretical reasons to believe otherwise.

Second: The fact that the AI has a “goodness” concept does not mean it is animated by that goodness concept, or animated by it in a robust way.

It’s one thing to get an AI to play a “good” role when it’s still weak enough to be playing whatever role it’s given; it’s a different thing altogether to get the whole morass of machinery and drives in the AI to be animated solely by a specific one of the AI’s concepts, even as the AI gets smarter and finds itself in drastically different contexts.^*

Modern AIs are the kinds of entities that can be lightly tuned one way and profess virtue, and lightly tuned another way and profess vice. An LLM is the kind of entity that fluidly swaps between personas; that talks a big game about ethics in one context and then does the opposite of what it says is ethical in other contexts. Recall how ChatGPT professes that psychotic people should not be egged on, and then eggs them on.

The critical question is what collection of drives animate the whole pile of machinery that make up the AI. Not just any one “mask” it sometimes wears, but the machinery that picks which mask to put forward.

Even if the AI had a “goodness” concept that was worthy of superintelligent pursuit, nobody has any idea how to grow an AI that robustly pursues a particular one of its concepts, much less an AI that pursues that concept and only that concept. Instead, we get AIs that are animated by a complex collection of drives that point who-knows-where.

Notes