When Does Recurrence Become an Algorithm?

Convergence Selection in Weight-Tied Looped Transformers
Tong Zhang1  ·  Junhao Hu2  ·  Yun Peng1  ·  Tao Xie2†
1Fudan University    2Peking University    †corresponding
📄 Paper (arXiv) BibTeX
A weight-tied looped transformer applies one block T times. Does loop t execute algorithm step t? Yes — exactly when the training budget demands one. The mechanism is a linear computation frontier whose speed is priced by the training budget, and the instruments the field uses to look for it are provably blind to it.

Overview

Overview: looped transformer, computation frontier, tau scaling classes
The measurement problem and the instrument. Standard probes (skip, cross-loop similarity, attribution) saturate at the fixed points trained loops converge to: they watch the tail of the loop trajectory while the algorithm lives in its head. The convergence-time scaling τ(n,i) reads the head. Its growth with input length is a mechanism signature (constant = shortcut, log n = parallel scan, linear n = serial), validated causally by activation-patching damage cones.

Four findings

1 · The budget law

Free training installs a frontier that solves v positions per loop, with v ∝ (ntr/Ttr)0.98±0.04, R²=0.99 across 23 models — exactly 1.00 under T=n training. The demanded minimum is met, never exceeded. Speed is a knob you set via the budget.

2 · Weight tying selects the mechanism

Depth-matched untied models do not learn the frontier at all: they stall near zero accuracy. Standard-depth transformers learn parallel scans on this family (Liu et al., 2023). The serial frontier is specific to the tied architecture.

3 · Walls are optimization objects

NC¹-completeness costs nothing (A₅ generalizes fully); group order does (S₅'s 120×120 operator deadlocks joint learning). An operator-first curriculum dissolves the wall in 10/10 seeds.

4 · Mechanisms are portable, not mandatable

Warm-starting across budgets transfers the algorithm in every seed and re-prices its speed. Imposing seriality through the input schedule fails where free training succeeds: streaming release trains parity perfectly (1.00, 8/8 seeds) yet scores 0.00 on A₅ (4/4) and S₅ (13/13).

The linear computation frontier

Frontier heatmaps
Per-(loop, position) accuracy at n=32. Each loop extends the solved prefix by a constant number of positions; the slope is a per-model fingerprint set by the training budget (v≈2.4 under tight log-T; exactly 1.0 under T=n; ≈2.1 under T=n/2).
Causal damage cones
The frontier is causal: resampling the state at (position, loop) produces strictly downstream damage whose cone slope independently reproduces each model's frontier speed.

The budget law & a halting rule

v  ≈  clip( ntrain / Ttrain,  vfree,  vmax )
Budget law fit
23 models across an 8× demand range: measured speed tracks demand with exponent 0.98±0.04.
Halting rule
The law is actionable: stopping at T* = ⌈n/v̂⌉ (v̂ read off training lengths) recovers near-oracle accuracy at test lengths and avoids overthinking.

Where the walls are

Walls track operator size, not complexity class
The matched-order contrast: A₅ (non-solvable, NC¹-complete) trains like the solvable groups; its complexity-class twin S₅ deadlocks on operator scale, and an operator-first curriculum dissolves the deadlock reliably.

Citation

@article{zhang2026recurrence,
  title  = {When Does Recurrence Become an Algorithm?
            Convergence Selection in Weight-Tied Looped Transformers},
  author = {Zhang, Tong and Hu, Junhao and Peng, Yun and Xie, Tao},
  year   = {2026},
  eprint = {2607.20594},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url    = {https://arxiv.org/abs/2607.20594}
}