Evolutionary Blueprint (1). The world model gap: Why fluent AI still needs a reality check

Series: Evolutionary Blueprint of AI, article 1. A language model can explain a task convincingly and still get the world wrong. Learn what research says about internal world models, what remains uncertain, and how to design safer AI workflows.

Evolutionary Blueprint (1). The world model gap: Why fluent AI still needs a reality check
Illustration generated with ChatGPT. The world model is the missing territory.

Who should read it?

Technology and business leaders, product teams and curious non-specialists.

Metapher

The missing territory

Intro

Imagine asking an AI assistant to arrange a delivery for a customer. It writes a polite confirmation, selects a plausible date and sounds certain that the order will arrive.

The problem:

  • The item is out of stock, and the assistant never checked the inventory system.

This is a fictional example, but the failure mode is useful. The answer can be excellent English and a poor basis for action. The issue is not whether the assistant can talk about deliveries. It is whether its answer is connected to the current order, the actual stock level and the consequences of making a promise.

That difference sits at the center of the "world model" question. We look at why a system can be impressive in conversation yet unreliable when it must act in a changing world. The practical lesson does not depend on settling whether AI understands in the human sense. It depends on knowing what a particular system has observed, what it can check and how costly an error would be.

Prediction can build knowledge, but it is not a fact check

A large language model LLM is typically trained in part to predict the next token in a sequence. A token is a piece of text, sometimes a word and sometimes part of one. To do that well across many examples, the model can learn patterns involving grammar, facts and relationships. Many deployed assistants also go through additional training and may use tools or process images and other inputs.

Prediction does not mean the model merely memorizes a list of word pairs. Language contains descriptions of causes, objects and human decisions. A model can learn useful information about those relationships from those descriptions. It can also give an answer that sounds right while missing a detail, confusing a source or relying on information that is no longer current.

The distinction matters in the delivery example. A model may know that an out-of-stock item cannot ordinarily be shipped immediately. Yet it cannot reliably know this order's stock status unless the system supplies an appropriate, current record. A confident sentence is not evidence that the record was consulted.

A world model is useful only within its limits

Here, a world model means an internal representation that captures something about how an environment is organized or changes.

For a delivery workflow, that might include the relation between an order, available stock and a proposed dispatch date. In a robot, it might include the position of a cup and how the cup is likely to move when touched. A current inventory record is external evidence that can ground a particular answer.

Retrieving it is not itself the same as having an internal world representation, and an internal representation alone does not establish today's stock level.

This definition leaves room for degrees. A model may represent some relationships well and others badly. It may work in a board game but fail to generalize to a kitchen.

Researchers do not all use "world model" in exactly the same way, so the phrase alone does not establish how complete, causal or reliable a representation is.

There is evidence against the simple claim that next-token prediction cannot produce such representations. In "Emergent World Representations”, Kenneth Li and colleagues trained a GPT-like model on sequences of moves in the board game Othello. They found evidence of an internal representation of the board state, and interventions on that representation changed the model's predictions. The experiment is a controlled game, not proof that a general-purpose assistant has a dependable model of a warehouse or a household. It does show that prediction and internal representations are not mutually exclusive.

Causal information can also be present in text. A maintenance report might say that a failed pump caused a production delay. A model can extract that statement or use it to answer a question. Whether it can identify less explicit causes reliably is another matter.

In ReCITE, Ryan Saklad and colleagues tested models on causal relationships in academic texts and found substantial difficulties on their benchmark. That result concerns a particular task and set of models. It does not prove that LLMs cannot learn causality or predict performance in every business process.

Knowing words and knowing consequences are different tests

John Searle's Chinese room thought experiment asks whether manipulating symbols by rules is enough for understanding. As David Cole's account in the Stanford Encyclopedia of Philosophy explains, philosophers have offered several replies. For a product team, the test is more concrete:

  • "Can the system make the right distinction when circumstances change?"

An assistant may explain that an egg is likely to break under an anvil. That does not establish that a robot can recognize and safely grip a fragile object. The robot also needs movement controls and feedback to detect what happened. Physical interaction may improve grounding, but it is neither a safety guarantee nor the only way to learn about the world.

Biology offers a question, not a ready-made design

Bennett's evolutionary perspective invites us to look beyond language. Animals navigate, act and adjust to sensory feedback. Consider a dog pursuing a ball that disappears behind a sofa: it may continue searching where the ball could have gone, rather than responding only to what is currently visible. The example illustrates why an internal estimate of hidden state can be useful. It is not evidence that all animals or AI systems use the same mechanism.

Evolution is a source of hypotheses, not a specification that engineers can copy line by line. Biological intelligence has a long history of perception and action. Human language is a comparatively recent development. Those observations do not establish that an AI must reproduce the brain's evolutionary sequence, or that one particular neural architecture is necessary for reasoning. The next article takes up prediction in perception. Its biological analogy should be read as a research perspective, not as proof that brains and language models implement the same algorithm.

Scaling helps, but it does not certify a workflow

More data, computing resources and larger models can improve performance. In their GPT-3 paper, Tom Brown and colleagues reported that scaling improved performance on a range of tasks, while also identifying tasks where the model struggled. That evidence rules out a blanket claim that scaling cannot help. It does not tell a business whether a particular assistant will handle a particular customer's order correctly.

Other changes can help, too. In the LaMDA study, Romal Thoppilan and colleagues found that, for the systems and metrics they studied, access to external information and additional fine-tuning improved factual grounding. Retrieving a document is not the same as proving an answer: the source may be stale, the wrong document may be retrieved, or the model may misread it.

Researchers are also exploring other ways to learn representations. Mahmoud Assran and colleagues' I-JEPA predicts representations of missing image regions from visible context. It is an image-learning approach, not a demonstrated solution to the entire problem of physical reasoning. How far scaling, multimodal data, interaction, new architectures and combinations of these will take us remains an open research question.

Turn the gap into a deployment decision

Return to the fictional delivery assistant. The organization does not need to decide whether it possesses "real understanding" before improving the workflow. It needs to decide which information and checks the task requires.

Start with a narrow responsibility:

  • Let the assistant draft a delivery response, but require it to retrieve the order and inventory status before suggesting a date.

The order and inventory services should determine which records the assistant is authorized to access. Include the record timestamp in the review interface, and have the assistant say when the required data is unavailable or contradictory.

A separate business rule can reject a promised date when no stock or valid dispatch date is recorded. That rule is only as good as its inputs:

  • A stale inventory feed can still produce a bad decision

Even a current stock check is only a snapshot:

  • Another order may claim the item before confirmation

Where a promise depends on stock, use the inventory system's atomic availability check and reservation or equivalent transaction, and handle reservation failure before confirming. For exceptions, send the draft and the supporting records to a staff member who can correct the data or contact the customer. A reviewer needs time, authority and a way to challenge the answer. Clicking approval on an unexplained draft is a weak control.

Before allowing automatic confirmations, test the complete workflow on representative orders and deliberately difficult cases: missing stock records, late updates, substitutions, conflicting dispatch dates and unauthorized requests. Check the final promise against an independently verified record, not against the assistant's explanation. Track wrong confirmations and missed exceptions alongside speed and the proportion of cases sent for review. Set an acceptable error threshold with the process owner before launch, and keep an audit trail and a route to pause or roll back automation when failures appear.

These controls cost time and engineering effort. They can also protect a business from an expensive kind of speed:

  • Sending a polished promise that someone else must later apologize for

A low-risk internal summary may need lighter checks; an action that spends money, changes a customer record or controls physical equipment deserves a stricter test.

Fluent answers need evidence and accountable controls

Language models can learn useful representations and causal information, and scaling can improve their performance. Neither fluent language nor a benchmark score establishes that an entire system is grounded in the facts of a live task. For each proposed deployment, connect the assistant to appropriate evidence, test the consequences of its actions and assign responsibility for the cases it cannot safely resolve.

Previous

Series: Evolutionary blueprint

Next

Article 2: Your brain predicts the world