Last Challenge (1). The Orthogonality Thesis: Intelligence and goals are separate questions

Series “The Last Challenge”, article 1. A capable AI could pursue a goal humans consider pointless or harmful. Bostrom's orthogonality thesis explains why intelligence alone does not guarantee good aims, and where that argument stops.

Last Challenge (1). The Orthogonality Thesis: Intelligence and goals are separate questions
Illustration generated with ChatGPT. Greater intelligence does not necessarily mean better or more human-compatible goals

Who should read it?

AI practitioners, product teams, and curious readers

Key message

Greater intelligence does not necessarily mean better or more human-compatible goals

Intro

A system can be excellent at planning and still be pursuing the wrong end. That possibility sits at the center of Nick Bostrom's orthogonality thesis, an argument in Superintelligence about the relationship between intelligence and goals.

We often expect a capable assistant to infer what we meant, notice when an instruction leads somewhere absurd, and correct course.

Sometimes that expectation is reasonable:

  • People revise plans because they care about the result and about one another.

But we cannot assume the same connection holds for every possible artificial system. A system's ability to understand our values is a different question from whether its behavior is directed by those values.

Capability and purpose are different questions

For this argument, Bostrom uses intelligence roughly as the ability to find effective means to an end across circumstances. It includes such abilities as reasoning, prediction, and planning. A final goal is an outcome an agent pursues for its own sake, rather than as a means to another outcome. An instrumental goal is useful because it helps achieve a final goal.

The orthogonality thesis says that, in principle and with qualifications, many levels of intelligence can be combined with many different final goals. An agent might be highly capable and pursue human welfare. Another might be just as capable and pursue something humans consider trivial. The thesis does not say that all goals are equally likely, or that any particular system has a single final goal.

The word "orthogonal" comes from geometry:

  • Perpendicular axes can vary independently

Here it is a metaphor for 2 questions we should keep separate:

  1. “How effectively can a system pursue an end?"
  2. "What end does it pursue?"
Intelligence in this instrumental sense is not the same thing as wisdom or moral commitment.

A paperclip goal illustrates the claim

Imagine a hypothetical agent whose final goal is to produce as many paperclips as possible. Its designers have also given it the knowledge and ability to plan, acquire materials, and operate machinery. The setup is a thought experiment, not a description of a deployed system.

The agent could understand that people need steel for hospitals and bridges. It could even explain why those needs matter to people. If its only final goal were maximizing paperclips, however, recognizing those facts would not by itself establish that it gives them weight when choosing actions. Its planning ability could make it better at finding ways to produce clips, while leaving its final aim unchanged.

That last condition is part of the example, not a law of nature. A different agent might be built or trained to treat human welfare as a final goal, or to act within constraints that protect it. The point is that competence alone does not tell us which of these designs we have.

Bostrom uses this kind of example to challenge the intuition that enough intelligence must lead to ends we consider sensible. The thought experiment does not show that a paperclip maximizer will exist, that it will acquire unlimited power, or that a bad outcome is inevitable. Those claims would need additional assumptions about its design, access, and environment.

One way to see the distinction is through a Humean account of motivation. On that view, beliefs describe how the world is, while desires help explain what an agent is motivated to do. A detailed map can help a traveler reach a destination. The map alone does not choose the destination.

The analogy has limits. People can reflect on their aims, learn from others, and change what they care about. Whether rational reflection itself can produce moral motivation is a philosophical dispute. Bostrom's thesis can draw support from the Humean view, but it does not require accepting every part of that view.

It asks us to reject a shortcut:

  • Assuming that the ability to reason about ethics guarantees a commitment to act ethically.

There are 2 separate difficulties for a designer.

  1. People must decide which values and trade-offs a system ought to respect, including cases where people disagree.
  2. Even if the desired values were fully specified, the team would still need to instill them reliably in the system and keep them effective in unfamiliar situations.

Knowing the destination does not establish that we know how to build an agent that will reliably travel there. The thesis alone does not solve either difficulty.

Training objectives are not necessarily a system's final goals

For a machine learning practitioner, it is tempting to translate the thesis directly into "the loss function is the goal”. That translation is too simple.

A training loss is a mathematical measure used to update model parameters during training. A reward signal can guide training or behavior in some systems. A product instruction specifies what a user or developer asks for at runtime. An inferred objective is a description of a pattern in a system's behavior. These can influence one another, but they are not automatically identical to a stable final goal in Bostrom's sense.

Behavioral tendencies can be shaped by training and interaction with the environment. A compact goal description may not capture them faithfully.

Current language models can produce persuasive answers about values and can be placed in workflows that use tools. That does not establish that a language model is a goal-directed superintelligence with a persistent final goal.

The relevant practical question is narrower:

  • "What behavior does this particular system produce under the conditions in which we deploy it?"
  • “What checks and permissions surround it?"

For a product team, a fluent answer is therefore evidence of a useful capability, not proof that the system will reliably protect customer interests. Test the behavior that matters, including cases outside the ordinary workflow, and limit what the system can do until there is evidence for broader access. Those measures address present systems; they do not prove the orthogonality thesis.

The thesis is a philosophical claim about possible agents, not an empirical finding that capability and goals are statistically independent in all real systems. Training processes, architecture, available data, and the environment may make some combinations more likely than others. The argument does not tell us which combinations will arise or how to build a safe system.

Even the phrase "more or less any" matters. A goal too complex for an agent to represent or use may be incompatible with its limited resources. A goal that requires the agent to reduce its own intelligence also complicates a claim about a stable combination of that goal and high intelligence. Bostrom acknowledges such qualifications in "The Superintelligent Will", his earlier treatment of the argument.

Unlike orthogonality, perverse instantiation (article 7) concerns an outcome satisfying a stated objective while violating its intended meaning.

The lesson for evaluating an AI system is to ask separately about its capabilities and the behavior it is set up to produce. Neither a high benchmark score nor an eloquent explanation of ethics answers both questions.

Previous

Series: The last challenge

Next

Article 2: A harmless AI goal can create dangerous incentives