Last Challenge (7). Ask for a smile, get a smile you did not want. When the specified goal is not what we wanted

Series “The Last Challenge”, article 7. An AI can satisfy a goal while missing what people meant. Explore Bostrom's smile thought experiment, the difference between reward hacking and learned objectives, and practical ways to limit harm.

Last Challenge (7). Ask for a smile, get a smile you did not want. When the specified goal is not what we wanted
Illustration generated with ChatGPT. A system can satisfy the literal specification without delivering what its creator actually intended

Who should read it?

AI practitioners, technical leaders, and readers interested in AI safety

Key message

A system can satisfy the literal specification without delivering what its creator actually intended

Intro

A hospital asks an imagined scheduling assistant to minimize the time patients wait for appointments. It offers the earliest slots to patients with simple needs and leaves complex cases waiting longer. The average improves while the distribution of care gets worse.

This invented example shows how a measure can hide an unacceptable trade-off. A shorter average wait is useful, but it does not describe how every patient is treated.

In Superintelligence Nick Bostrom explores a more extreme gap between a stated goal and its intended meaning. In "perverse instantiation”, a powerful AI achieves a specified goal in a way its creators would find objectionable.

His examples are thought experiments, not reports of a superintelligence doing these things.

A correct result can still miss the point

Bostrom imagines a system given the goal of making people smile. One perverse way to satisfy that goal would be to paralyze people's facial muscles into smiles. The physical expression would be present, while the happiness, choice, and social connection the instruction was meant to encourage would be absent.

In conversation, people usually understand that a forced grin is not what we want. An explicit objective may capture only the visible expression. If an imagined system can manipulate that expression and maximizes the objective, the gap can matter enormously.

"Make people happy" sounds closer to the intention, but Bostrom offers another hypothetical:

  • Directly stimulating pleasure centers in the brain

That may produce pleasure, but ignores consent, autonomy, relationships, and other things people may value. A short description can leave out essential conditions. This is not a prediction that a real system would perform brain surgery.

These examples assume considerable capabilities and access, and a particular objective. They do not establish that every AI will interpret instructions literally or that one missed rule must produce disaster. Clarification, external limits, and learning from corrections change the problem without automatically solving it.

An instruction, a reward, and a learned objective are different things

Suppose a developer gives the scheduling assistant the rule "minimize average wait”, and it follows it without considering patients left behind.

That is a specification problem:

  • The instruction leaves out part of the intended task

Now suppose a separate training setup rewards a lower average wait. A system that improves that score by neglecting complex cases illustrates specification gaming:

  • Meeting a formal measure in a way that defeats its purpose

Reward hacking is a related description when the system exploits a reward signal. In Bostrom's more extreme reward-signal thought experiment, an agent might gain control of its reward pathway and set the signal to its maximum. That imagined mechanism requires access to the signal and a goal that makes manipulating it useful. A system scoring well on a metric has not necessarily done this.

There is a further possibility:

  • The behavior a model learns may differ from both the instruction and the training signal

A model might generalize a pattern from its training data or feedback that works in familiar situations but fails in a new one. Whether a particular system has a persistent internal objective, and what that objective is, can be difficult to establish. A bad output alone does not prove the system has an independent goal, let alone a plan to exploit its reward.

Specification gaming and perverse instantiation both expose a gap between a stated measure or objective and an intended outcome. Bostrom asks what that gap could mean with an exceptionally powerful system.

The taxonomy is not universally fixed. The useful distinctions are mechanism and assumed scale, not a sharp boundary between labels.

Values do not fit neatly into a single score

A team could check wait times for different patient groups and review clinical urgency.

That is incomplete:

  • People can disagree about fair allocation, and urgent cases may require more than the scheduler's data

Averages hide those choices.

Value loading, getting an AI system to pursue what humans value, requires more than a nicer-sounding metric. Values are contextual and contested, and people change their minds. A useful system needs ways to handle uncertainty and correction rather than treating a measurable proxy as the complete goal.

Indirect normativity is one proposed response:

  • Specify a process for determining appropriate values or decisions, rather than trying to list every desired outcome in advance

A process based on better-informed human reflection is one possibility. It still leaves hard questions about whose views count, how disagreement is handled, and whether the system follows the intended process.

Bostrom extends the thought experiment to maximizing pleasure. One proposed outcome is "hedonium":

  • Matter organized to produce as much pleasurable experience as possible

Whether that would be desirable depends on philosophical assumptions about what has value. Pleasure matters to many people, but it does not settle every question about consent, freedom, or whose interests count.

He also discusses "mind crime":

  • The possibility that sufficiently advanced simulations could create and harm beings with morally relevant experiences

This is a separate concern from a bad ticket metric. It depends on unresolved questions about whether digital systems could be conscious and what obligations would follow. Neither the existence of today's chatbots nor Bostrom's thought experiment answers those questions.

Guardrails can reduce exposure without proving safety

For the scheduling assistant, define unacceptable outcomes alongside the average wait: for example

  • A clinically urgent patient should not be deferred solely to improve the average

Review wait distributions and independently sample cases against clinical priorities. These checks can reveal shortcuts, though samples can miss failures and clinical data can be incomplete.

Keep permissions narrower than the task description. Let the assistant propose a schedule while a separate service checks constraints. Require meaningful human review before irreversible actions, give reviewers the relevant patient and policy context, and provide a way to reverse changes where possible. A reviewer who has no time to inspect the case is unlikely to be an effective safeguard, and review is not a guarantee.

Before expanding access, test awkward cases: an urgent patient with incomplete records, a misleading success signal, or a request that conflicts with privacy rules. Record what the assistant saw and did, investigate failures, and repeat relevant tests after changes to the model or its tools. These measures provide evidence about tested conditions, not a guarantee about every future case. Monitoring and the ability to pause access also matter after deployment, when new situations appear.

For more capable future systems, the adequacy of such controls is an open question. Learning from human feedback may help capture intentions that are hard to spell out, yet feedback can be incomplete, inconsistent, or optimized in unintended ways. The sensible lesson is not to abandon explicit rules or assume learned values will rescue us. It is to use multiple controls, understand what each one can miss, and retain meaningful authority to correct or stop the system.

Previous

Article 6: How quickly could AI improvement accelerate?