Last Challenge (6). How quickly could AI improvement accelerate? The takeoff might not give us a weekend to react.

Series “The Last Challenge”, article 6. AI systems might improve gradually or in a rapid burst. Learn what could drive each scenario, why an exponential curve does not go vertical, and which practical limits make predictions uncertain.

Last Challenge (6). How quickly could AI improvement accelerate? The takeoff might not give us a weekend to react.
Illustration generated by ChatGPT. If AI capabilities accelerate rapidly enough, the time available to recognize and respond to a critical transition may be much shorter than expected.

Who should read it?

AI practitioners, technical leaders, and readers interested in AI safety

Key message

If AI capabilities accelerate rapidly enough, the time available to recognize and respond to a critical transition may be much shorter than expected.

Intro

A team lets an AI research assistant suggest changes to its software. Most suggestions are useless, but a few speed up experiments. If later versions propose better improvements,

"Could progress leave people little time to respond?"

This invented scenario captures the question behind AI "takeoff":

"How quickly might a system reach a much higher level of capability?"

Nick Bostrom examines possible futures, not a measured forecast of when or how a transition will happen.

Soft takeoff vs. hard takeoff

A soft takeoff describes a comparatively gradual transition, giving developers and institutions more time to respond.

A hard takeoff is much faster. The labels are relative, not precise units:

  • Speed depends on the capabilities compared and the time available to react

Feedback can speed up research, but it has to work

Recursive self-improvement is the idea that an AI system might contribute to improving the process that produces more capable AI systems. An assistant might propose a better training method, help diagnose a failed experiment, or write code that researchers test in a successor system. If those improvements make the next round of research more productive, the process can form a positive feedback loop.

Crossover point

The crossover point is the hypothetical point at which the system's own contribution to its improvement overtakes the external contribution, such as human research effort. Passing it would not prove that improvement accelerates without limit:

  • The combined effort still has to overcome the difficulty of finding and testing the next improvement

A suggestion must be implemented and tested, its result checked, and the improved system made available for the next round. A model writing its own code does not automatically know how to become more capable. It can introduce errors, optimize a misleading benchmark, or propose changes too expensive to test.

The loop also depends on what the system can access. Improving a training algorithm may require compute, data, engineering time, and evaluation. Improving chips may require manufacturing capacity that software alone cannot summon. Some research tasks can run in parallel. Others require sequential experiments or human decisions. A capable system might reduce some of these delays, but the size of that benefit is an open question.

If improvements arrive faster than a team can evaluate and deploy them, pressure to release poorly understood systems could mount. Costly experiments and gains that are difficult to verify could instead slow progress. The outcome depends on more than whether AI contributes to research.

More effort may meet harder problems

Bostrom presents a conceptual relationship between the rate of improvement, optimization power, and recalcitrance:

  • More effective effort tends to increase the rate, while greater resistance to improvement reduces it

Optimization power is the effective effort directed at improving a system. It can include human research and, in the scenario under discussion, useful work contributed by AI.

Recalcitrance describes how resistant the system is to improvement:

  • High recalcitrance means that a given amount of effective effort buys less progress

A team running more useful experiments each week has increased its optimization power. If easy changes have been found and remaining gains require harder experiments, recalcitrance has increased too. More effort alone does not tell us how fast capability will rise.

This is a conceptual model, not an operational forecast. "Capability" is difficult to compress into one number, and the terms lack generally agreed measurement scales. The factors can change with the task, resources, and stage of development. The relationship frames questions without supplying values.

Exponential growth can be very fast, but in a standard mathematical model it remains finite at every finite time. It does not by itself have a finite-time vertical asymptote.

Different assumptions can produce a finite-time blow-up in a mathematical model, such as a sufficiently strong nonlinear relationship between capability and its rate of change. That result would follow from those assumptions, not from exponential growth itself. It would still not establish that a real AI system can bypass limits on hardware, experiments, or deployment.

Extra hardware could help, but an overhang is not assured

A hardware overhang is a possible gap between available computing capacity and what existing AI software can use effectively. If a later software advance could make use of capacity that was already built, capability or deployment might rise more quickly than it would if everyone had to wait for new chips. That is a scenario, not a guarantee that such spare capacity will exist when it is needed.

Existing computers are not interchangeable free resources. They may already serve customers, lack the memory or networking a system needs, or be in places where power, access, or cost constrains their use. Running many copies also requires enough computing capacity per copy. More copies can help with parallel work, but they do not automatically speed up every sequential experiment, and coordinating their output can create new bottlenecks.

A business claiming a breakthrough can immediately scale needs a capacity plan covering existing workloads, new requirements, and switching time. Data centers alone do not establish available capacity.

Speed changes the response window, not the winner by itself

Slow transition

In a slow transition, researchers, competitors, regulators, and customers may have more time to observe progress and adjust. That extra time is useful only if they can recognize the relevant changes and have the incentives and authority to act. A gradual transition can still produce harmful deployments or leave institutions behind.

Fast transition

In a fast transition, evaluation and oversight could struggle to keep pace. Bostrom considers whether a leading project might obtain a decisive strategic advantage, enough power to overcome effective opposition. That outcome would require more than being first: the lead would have to translate into durable real-world advantages despite competitors, resource constraints, and other forms of resistance. A fast takeoff does not establish winner-take-all, and a slower one does not guarantee a balanced distribution of power.

Singleton

A singleton is a world order with a single decision-making agency at the highest level. A decisive strategic advantage could contribute to such an outcome under further assumptions about lasting control and governance. Neither a singleton nor its character is an established consequence of a fast takeoff.

A practical way to prepare without pretending to know the future is to define a review trigger before deploying a more capable system. For example, a team could require a fresh evaluation when an assistant gains access to new tools, can run research tasks for longer, or contributes to changes in its own development pipeline. The review should identify who can pause deployment, what evidence is needed, and which permissions remain outside the assistant's control. Those controls will not settle the takeoff debate or guarantee safety, but they make the response less dependent on improvisation during a release.

Previous

Article 5: Could a brain be reproduced in software?

Next

Article 7: When the specified goal is not what we wanted