Transformers are an interacting particle system, and their tokens cluster.
Softmax needs exp, which this tree does not have — so replace the kernel
with a rational one and check, first, whether the phenomenon comes with it. The PATH 12 rule:
when the arithmetic cannot represent your model, change the model, do not approximate the function.
Softmax attention needs exp. The PATH 12 move
is to substitute a rational kernel rather than approximate the exponential. The deciding
question: does the clustering phenomenon survive the swap? If not, the certificate would
describe a system nobody cares about.
It survives, and it survives structurally. Measured on 12 tokens on S²,
fixed seed (attention-substitution.py):
| kernel | low sharpness | mid | high |
|---|---|---|---|
| softmax — blocked | 1 cluster | 1 cluster | 3 clusters |
polynomial (1+⟨xi,xj⟩)p | 1 cluster | 1 cluster | 2 clusters |
Both undergo a cluster-count change as the interaction sharpens. Green control: if softmax ever fails to cluster the script exits 2.
python3 research/_frontier/attention-substitution.py
An integer exponent is a poor bifurcation parameter. The form that stays rational in the state and smooth in the knob:
wij ∝ (1 + β⟨xi, xj⟩)p
with integer p and continuous β —
only + − × ÷ and integer pow.
The projected residual is exactly zero on closed-form
coincident-cluster configurations, decided by exact rational arithmetic.
Antipodal 2-clusters and ⟨u,v⟩ = −1/β 2-clusters
(when β ≥ 1) are equilibria for every such β.
Existence of multi-cluster equilibria is not what any float β* measured.
What would falsify the enclosure. A nonzero exact residual on the closed-form configurations (T1–T3). A zero residual on a deliberately wrong angle (T4 must stay red). Krawczyk closing on a non-isolated coincident root (T5 must refuse).
Locator transient. attention-transition.py reported
β* ∈ [1.601593, 1.601624] from 4000-step terminals.
That bracket collapses under Newton / 20k steps. Gated in
attention-equilibrium.py T5. Same species as PATH 12's first Huggett test.
Newton-from-8k is not an omega-limit.
attention-equilibrium.py locked β* ∈ [1.615215, 1.615220]
by short flow + Newton. Near criticality that is metastability.
Gated as T8 in attention-basin.py: 100k-step flow from seed +0 at the old
βhi reaches the 1-cluster.
(2.5, 2.7) was an 80k-step cutoff, not a linear crossing.
The old basin T6/T7 reported the exact 2-cluster “loses local stability in (2.5, 2.7)”
because β=2.5 still showed 2 clusters at 80k steps and β=2.7 showed 1. Extending the
budget: the same seed/eps collapses at ~84.5k for β=2.5 (gated T9). Exact rationals
(attention-stability.js) decide the reduced perfect-cluster law has a
double zero at c*=−1/β and ċ>0 off equilibrium
for every tested β — one-sided semi-stable, no linear sign change in β.
attention-enclosure.js — exact rational
arithmetic (eqcert/src/rational.js) decides the residual is all zero on:
| configuration | β | residual |
|---|---|---|
2-cluster, ⟨u,v⟩=−3/5, Pythagorean v=(−3/5,4/5,0) |
5/3 | exactly 0 |
| antipodal 2-cluster | 1/2 (below any float bracket) |
exactly 0 |
| 1-cluster | 5/3 | exactly 0 |
Claim revision, gated by the antipodal β=1/2 case:
multi-cluster equilibria exist far below any float bracket. Krawczyk on an ungaged coincident
root refuses (geometric non-isolation).
node research/_frontier/attention-enclosure.js
attention-spectrum.py. At any 1-cluster
equilibrium, every sphere-tangent perturbation satisfies
(Jv)i = mean(v) − vi
— spectrum {0 (gauge), −1 (relative)},
independent of β. At consensus the weights are uniform and their
first variation vanishes.
Therefore no float β* on this path is a linear bifurcation
of the 1-cluster. The 1-cluster stays linearly attracting for all β.
python3 research/_frontier/attention-spectrum.py
attention-basin.py. At rational
β = 5/3, both closed-form equilibria are locally stable under the
normalized flow (50k steps after a 10−3 perturbation):
| start | after perturbation + flow |
|---|---|
| 1-cluster | 1 cluster |
exact 2-cluster ⟨u,v⟩ = −3/5 | 2 clusters |
From seed +0, 100k-step flow selects 1 cluster at β = 3/2
and 2 clusters at β = 2. Perfect-cluster kicks: above c* merges,
below returns — at both β=2.5 and 2.7.
A tight certified omega-limit β* for seed +0 is not claimed
(critical slowing).
python3 research/_frontier/attention-basin.py
attention-stability.js — exact
rationals decide the perfect-cluster reduced law (equal sizes, p=2):
ċ = 2(1+βc)²(1−c²) / [(1+β)² + (1+βc)²]
Double zero at c*=−1/β (so f=f′=0);
ċ>0 off equilibrium on every tested rational point —
one-sided semi-stable for every β, not a linear crossing in (2.5, 2.7).
Cross-weights are flat to first order at p=2, so the tangent Jacobian
block-decouples into two 1-cluster copies with spectrum {0,−1}.
node research/_frontier/attention-stability.js
| claim | status | what backs it |
|---|---|---|
| rational kernel still clusters | measured | attention-substitution.py, exit 0 |
| exact residual zero on closed forms | measured | attention-enclosure.js, T1–T3 exact rationals |
| 1-cluster spectrum {0, −1}, β-free | measured | attention-spectrum.py, analytical J = mean − Id |
| bistability at β=5/3 | measured | attention-basin.py T1–T3 |
| reduced 2c angle: one-sided semi-stable | measured | attention-stability.js T1–T7 exact rationals |
| tight omega-limit β* for seed +0 | open | not claimed; Newton bracket demoted (T8) |
| (2.5, 2.7) linear stability loss | demoted | 80k cutoff; T9 + double-zero law |
| Transformer / softmax threshold | never | rational system only; must not be reported as softmax |
Second independent instance of the PATH 12 rule. PATH 11 died approximating; PATH 12 worked by substituting in economics; this substitutes in machine learning. Two confirmations and one refutation is what turns a trick into a method.
It takes attention-bifurcation (CONFIRMED in
SCORING.json) out from behind interval-transcendentals.
PATH 07's consumer count drops again.
Harder swap than PATH 12. Huggett is still a canonical model. Linear / polynomial attention is a different attention; Geshkovski et al. is about softmax. The certified number would be a fact about the rational system and must never be reported as a fact about Transformers.
Evidence is still thin, and under-iteration already bit thrice.
Bistability is gated at one rational β; seed +0 selection at two
safe distances. Never trust a cluster count that has not been long-integrated or
Newton-polished to a small algebraic residual.
Occupancy, ungated. Geshkovski, Letrouit, Polyanskiy and Rigollet are named from memory; Lessard–Pugliese likewise. N is near zero and the artifact is the claim — no literature gate has run.
N: near ZERO. Interacting-particle views of
attention are published; certified bifurcation brackets are standard machinery.
A: HIGH and unverified — a re-runnable, zero-dependency artifact for an
attention system does not appear to exist (same A-axis claim as
attention-bifurcation in SCORING.json).
Citation state: Geshkovski et al. and Lessard–Pugliese are named from memory and unverified at source. The measurements above were produced on 2026-07-31 by the scripts named, and are reproducible with a fixed seed.