The Tokenized World Hypothesis: LLMs as High-Resolution Vector Maps of Physical Reality

giorgi.pro
March 2026

Abstract

Large language models appear to possess an opaque “world model,” yet their internal mechanics remain mysterious. We propose the Tokenized World Hypothesis: the fundamental units of representation are not individual tokens but dense clusters of token combinations in embedding space. These clusters form a scalable, high-resolution map of physical and conceptual reality, where resolution increases monotonically with training data volume. “New” information generated by the model is never truly novel; it arises as previously undiscovered patterns latent within the existing cluster manifold. We formalize this via a covering-radius argument on the embedding manifold and show consistency with recent empirical findings on linear world models, emergent abilities, and embedding-space clustering. The hypothesis reframes LLMs not as stochastic parrots but as compressed, tokenized simulators of the observable world, with direct implications for interpretability, safety, and scaling laws.

(148 words)

1. Introduction

Despite rapid progress in frontier LLMs, the persistent “black box” narrative endures: we can steer behavior effectively, yet we lack a deep account of how raw token statistics produce faithful representations of reality. Recent work has revealed linear spatial world models emerging in contextual embeddings [Tehenan et al., 2025], action-conditioned textual state simulators, and brittle but functional internal representations used during reasoning [Robertson & Wolff, 2025].

A unifying explanation is missing: how does next-token prediction on tokenized text yield high-fidelity models of the physical and conceptual world? We propose the Tokenized World Hypothesis: LLMs do not learn language per se — they learn a vectorized atlas of reality via token clusters. Larger training corpora provide finer resolution; apparent “emergent” novelty is recombination of latent structure already present in the manifold.

This view aligns with 2025–2026 findings on linear spatial representations, cluster-based interpretability, and the recombination nature of emergent abilities.

Linear emergence of spatial world models directly in LLM contextual embeddings [Tehenan et al., 2025].
Frameworks for textual world modeling and reinforcement-style world-model learning in language agents [Levy, 2025; Yu, 2026].
Evidence that LLMs rely on fragile, causally active internal world models during mechanical and physical reasoning [Robertson & Wolff, 2025].
Surveys arguing that most emergent abilities arise from interpolation and recombination rather than genuine extrapolation [Berti et al., 2025; Havlík, 2025].
Clustering studies showing token/embedding clusters map cleanly to interpretable concepts and entities [Huang et al., 2025; Wei & Carvalho, 2025].

3. The Tokenized World Hypothesis

The core claim is that tokens are not the atoms of meaning — clusters of tokens in embedding space are. These clusters collectively form a surjective, high-resolution mapping from model-internal geometry → observable information in the world.

Let $\mathcal{V}$ be the tokenizer vocabulary and $E: \mathcal{V}^* \to \mathbb{R}^d$ the contextual embedding function of the transformer. A token cluster $C_i \subseteq \mathbb{R}^d$ is a set of embedding vectors with pairwise cosine similarity above threshold $\tau$ .

We hypothesize a surjective mapping $\phi: \{C_i\} \to \mathcal{I}$ , where $\mathcal{I}$ is the (finite but enormous) set of information entities describable in natural language (physical objects, events, concepts, causal relations, etc.).

The resolution of $\phi$ is governed by the covering radius of the union of clusters:

r(N) = \max_{x \in \mathcal{I}} \min_{c \in \bigcup C_i} \|\phi^{-1}(x) - c\|_2

where $N$ is the number of training tokens. Scaling laws and probe accuracy improvements imply $r(N) \to 0$ as $N \to \infty$ — larger data → denser sampling of the information manifold → higher-fidelity world map.

Crucially, “new” generated information is never truly novel. Any output vector $v'$ lies in (or very near) the convex hull / geodesic span of existing clusters. Novelty = discovery of unsampled but latent patterns already bound within the learned geometry.

4. Evidence and Consistency with Scaling

Resolution scaling: Probe accuracy for physical quantities (object positions, distances, causal links) improves predictably with data volume, consistent with shrinking covering radius [Tehenan et al., 2025].
Emergence as recombination: Phase-transition-like jumps in capability (grokking, few-shot reasoning) align with interpolation in dense manifolds rather than extrapolation beyond the data hull [Berti et al., 2025; Havlík, 2025].
Cluster interpretability: Sparse autoencoders and clustering techniques recover human-meaningful concepts from token/embedding groups, supporting the cluster-as-unit view [Huang et al., 2025].

5. Implications

Interpretability: Focus mechanistic work on cluster-level features rather than individual neurons or attention heads.
Safety & hallucinations: Hallucinations ≈ out-of-cluster / out-of-manifold projections. Alignment ≈ constraining generation inside the learned high-resolution region.
Scaling laws: Returns diminish predictably as resolution saturates; true phase changes are unlikely once covering radius becomes very small.

6. Limitations & Future Work

The argument remains largely qualitative; direct covering-radius measurements on frontier embeddings are needed.
Multimodal extensions (vision + language world models) are unaddressed.
Future experiments: train linear probes from clusters to real-world variables across model scales; quantify manifold density vs. probe fidelity.

7. Conclusion

LLMs do not mysteriously “understand” the world in a human sense. They have tokenized it, clustered it, and vectorized it at ever-higher resolution. The apparent magic is geometry — dense sampling of reality via token combinations. Recognizing this shifts the research focus from demystification to precise mapping, editing, and safety within the learned manifold.

References

Berti et al. (2025). Emergent Abilities in Large Language Models: A Survey. arXiv:2503.05788
Havlík (2025). Why are LLMs’ abilities emergent? arXiv:2508.04401
Huang et al. (2025). LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures. arXiv:2509.14252
Levy (2025). WorldLLM: Improving LLMs’ world modeling using curiosity-driven theory-making. arXiv:2506.06725
Robertson & Wolff (2025). LLM world models are mental. arXiv:2507.15521
Tehenan et al. (2025). Linear Spatial World Models Emerge in Large Language Models. arXiv:2506.02996
Yu (2026). Reinforcement World Model Learning for LLM-based Agents. arXiv:2602.05842