Isn’t it impossible to predict the behavior of a superintelligence?
In some respects, but not in every respect.
Stockfish 17 is better at steering a chessboard than we are. If we played a chess match against Stockfish, we would not be able to predict its moves — doing so would require us to be at least as good at chess as Stockfish 17. But it would be easy for us to predict the winner of the match.* It’s hard to predict what moves Stockfish will make; it’s easy to predict that it will win.
So too with AIs that are predicting and steering the real world. The smarter they are, the harder it is to predict exactly what they do, but the easier it is to predict that they’ll reach whatever destination they were steering toward.
* This answer indicates an interesting epistemological state. When you believe that Stockfish is smarter than you at chess, your beliefs about the final outcome of the chess game are not fully captured by your best predictions about Stockfish’s individual moves.
A philosopher of science might ask how that can possibly be the case, when the rules of chess are fully known and the outcome follows exactly from each exact move. The answer is that there’s a vast logical structure of possible chess games, which in one sense is fully determined by the chess rules, but which is not fully known to you (nor even to Stockfish!) because your mind cannot visualize all the consequences that follow from the premise of the chess rules.
A “smarter” chess-player can be viewed as knowing relatively more truths about this chess-possibility-space than you do; so when you see a surprising-to-you move from a “smarter” chess-player, that implies a new-to-you fact about those unknown consequences of the known rules of chess, which in turn weighs on your expectations about the game’s outcome.
(One would expect the previous paragraphs to be a standard idea in computer science. To our surprise, it’s not. Most of computer science, and indeed most of academia up until this point, has not been terribly interested in any ideas associated with superhuman intelligence.)