When Does Recurrence Become an Algorithm?

Convergence Selection in Weight-Tied Looped Transformers
Tong Zhang1  ·  Junhao Hu2  ·  Yun Peng1  ·  Tao Xie2†
1Fudan University    2Peking University    †corresponding
📄 Paper (PDF) 🤗 Hugging Face BibTeX
A weight-tied looped transformer applies one block T times. Does loop t execute algorithm step t? Yes — exactly when the training budget demands one. The mechanism is a linear computation frontier whose speed is priced by the training contract, and the instruments the field uses to look for it are provably blind to it.

Overview

Overview: looped transformer, computation frontier, tau scaling classes
The measurement problem and the instrument. Standard probes (skip, cross-loop similarity, attribution) saturate at the fixed points trained loops converge to — they watch the tail of the loop trajectory while the algorithm lives in its head. The convergence-time scaling τ(n,i) reads the head: its growth with input length (flat / log / linear) is a mechanism signature, validated causally by activation-patching damage cones.

Four findings

1 · The budget law

Free training installs a frontier that solves v positions per loop, with v ∝ (ntr/Ttr)0.98±0.04, R²=0.99 — exactly 1.00 under T=n training. The demanded minimum is met, never exceeded. Speed is a knob you set via the contract.

2 · Architecture picks the algorithm

Standard-depth transformers learn parallel scans on this task family; weight tying flips the selection to the serial frontier. At matched depth and parameters, untied models extrapolate worst — and fail to learn A₅ at all.

3 · Walls are optimization objects

NC¹-completeness costs nothing (A₅ generalizes fully); group order does (S₅'s 120×120 operator deadlocks joint learning). An operator-first curriculum dissolves the wall in 10/10 seeds.

4 · Mechanisms are portable, not mandatable

Warm-starting across budget contracts transfers the algorithm in every seed and re-prices its speed; imposing seriality through the input schedule fails where free training succeeds.

The linear computation frontier

Frontier heatmaps
Per-(loop, position) accuracy at n=32. Each loop extends the solved prefix by a constant number of positions; the slope is a per-model fingerprint set by the training contract (v≈2.4 under tight log-T; exactly 1.0 under T=n; ≈2.1 under T=n/2).
Causal damage cones
The frontier is causal: resampling the state at (position, loop) produces strictly downstream damage whose cone slope independently reproduces each model's frontier speed.

The budget law & a halting rule

v  ≈  clip( ntrain / Ttrain,  vfree,  vmax )
Budget law fit
23 models across an 8× demand range: measured speed tracks demand with exponent 0.98±0.04.
Halting rule
The law is actionable: stopping at T* = ⌈n/v̂⌉ (v̂ estimated in-distribution) is near-oracle and avoids overthinking collapse.

Where the walls are

Walls track operator size, not complexity class
The matched-order contrast: A₅ (non-solvable, NC¹-complete) trains like the solvable groups; its complexity-class twin S₅ deadlocks on operator scale — and an operator-first curriculum dissolves the deadlock reliably.

Citation

@article{zhang2026recurrence,
  title  = {When Does Recurrence Become an Algorithm?
            Convergence Selection in Weight-Tied Looped Transformers},
  author = {Zhang, Tong and Hu, Junhao and Peng, Yun and Xie, Tao},
  year   = {2026},
  note   = {Preprint}
}