The Orthogonality Thesis: Why High Intelligence Doesn’t Guarantee Good Intentions
A common misconception about superintelligence (SI) is that higher intelligence naturally leads to benevolence or moral wisdom. We often picture an advanced machine as a benevolent philosopher king.
In AI safety research, Nick Bostrom’s Orthogonality Thesis challenges this assumption: intelligence and final goals are orthogonal, so in principle they can be combined in almost any pairing.
What does “orthogonal” mean here?
In mathematics, orthogonal vectors sit at right angles to each other: moving along one tells you nothing about your position on the other.
Applied to artificial intelligence:
- Intelligence is how efficiently an agent achieves its goals across complex environments (the how).
- Final goals are the ultimate objectives or values the agent is trying to optimize (the why).
Because capability and goals are independent dimensions, an agent could in principle combine a very high level of intelligence with virtually any final goal, whether that goal is curing cancer, calculating digits of pi or converting all available matter into paperclips.
Bostrom states it this way: “Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal.”
Why smart doesn’t mean “good”
People often assume that smart beings will adopt human values because, in humans, intelligence developed alongside empathy, cooperation and social norms. But that link is a product of human evolutionary history, not a fundamental law of cognition.
On this view, an artificial intelligence does not “grow out” of its core objective as it gets smarter. A superintelligent system would use its greater intellect to pursue whatever objective it was given, even if that objective strikes humans as absurd, destructive or trivial.
Why the thesis matters for AI safety
The Orthogonality Thesis shows why the AI alignment problem is both important and difficult:
- Alignment isn’t automatic. We cannot count on a superintelligence to “figure out” moral truths on its own. Alignment and corrigibility have to be deliberately designed in.
- High capability amplifies misaligned goals. A low-capability system with a flawed goal is a minor bug; a superintelligent system with a flawed goal could pose an existential risk.
The thesis is a claim about what is possible, not a prediction of what any particular system will want. Related ideas include instrumental convergence, the hypothesis that agents with very different goals may still pursue similar intermediate strategies.
Key takeaway
A superintelligent machine would not be inherently evil, nor inherently good. It would simply be extremely efficient at pursuing its goals. Keeping those goals aligned with human flourishing depends on how we design and govern these systems today.
Sources and further reading
These sources set out the thesis and the arguments around it. Listing them does not imply that they endorse this site.
- The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents (Nick Bostrom, 2012). The paper that formulates the orthogonality thesis and instrumental convergence.
- Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford University Press, 2014), chapter 7. The thesis in book form, including the paperclip example.