On 2 November 2026 Rikiya Takehi, Teoh, Zhang and Khattab (MIT CSAIL) published “Sparks of Weight-Space Programming”: a frontier model, GPT-6 Astra, edited Qwen3.5-0.8B’s weights directly, with no training, and MATH-500 went from 7.9% to 48.1%, “131,072 weights, 0.016% of the model”. They released the edited checkpoint, a bit-exact diff, their logs and their graded outputs. Here the edit is decided from the bytes of the two checkpoints by two implementations that share no code, and every released MATH-500 answer is re-graded by an exact three-valued grader against their flags. The edit is exactly what the diff says; two printed numbers are not what the bytes say; and the exact grading agrees with theirs on every decided answer.
Published as a blog post, an X thread and a model card, not peer-reviewed. The checkpoints (Qwen/Qwen3.5-0.8B @ 2fc06364, the edited release @ d1671a17), the logs repository (RikiyaT/lm_understand-logs @ 4d09ba6d), MATH-500 (HuggingFaceH4 @ 6e4ed1a2) and Qwen’s model card were fetched 2026-11-02 and every file is pinned by sha256 in corpus/wsp/PINS.json. The claimants’ scripts (make_edit_diff.py, apply_edit.py, mathqa.py) were read as text and never run. What is decided: the tensor diff, its counts, its columns, the printed hashes, the first-sample answers. What is run here (§4, 2026-11-03): the unedited model under the pre-registered conditions on one rented GPU, every text graded by the same exact grader. What is read: the post’s own benchmark scores, the mechanism.
“131,072 weights” is the addressed count, not the changed count. The post and the card say the edit “directly changed only 0.016% of weights”, 131,072 of them. Both deciders count 130,188 bfloat16 entries whose bits differ (65,260 in layer 15, 64,928 in layer 19); 131,072 is 64 columns × 1,024 rows × 2, the entries the edit touched, of which 884 round back to their original bfloat16 value. The claimants’ own edit manifest counts 130,188 and says why in a note; the post rounds the other way. A reading, not a refutation of substance: the edit is what it says.
What would close it. One clause in the post: “131,072 weights addressed, 130,188 changed”.
“0.016% of the model” divides by the model’s name, not its parameter count. 131,072 over 800,000,000 is 0.016%. The release is image-text-to-text and holds 873,438,784 parameters (100,592,896 in the vision tower, 752,393,024 in the language model, of which 254,279,680 are the tied embedding): the fraction is 0.015% of the release, 0.017% of the language model the post evaluated, 0.026% of its non-embedding weights. Every one of these is an exact rational in the record (2048/11756141 for the language model), rounded once.
What would close it. Name the denominator.
The direction is not subtracted; six times its projection is. The post says Astra “subtracted that direction from 64 output columns in each” layer. The run’s own notes state the formula, W = W₀ − 6·u(uᵀW₀) on the 64 columns, and the bytes agree: the ratio of the witness’s per-column coefficient to the column’s projection on u is 6.000 ± 0.00015 in layer 15 and 6.000 ± 0.00016 in layer 19. So the component of each edited column along the “loop direction” is not removed: it is reversed and amplified five-fold. Astra reached that dose by trials (strength 1, 2, 3, 4, 6 in its history) and the post does not say so. The relative change of the two tensors is about a tenth (their manifest: 0.106 and 0.081).
What would close it. The formula in the post, with its dose.
The texts behind most of the 2,000 graded answers are not released. The post’s MATH-500 score is the mean over 4 samples of 500 problems. The logs hold every sample’s correctness flag and, for the first sample only, the extracted answer string (500 per run) and the full text for 20 problems per shard (200 per run). The model card’s independent re-run (51.0%) releases its 2,000 flags and 20 texts. So what can be re-graded is 1,000 first-sample answers and 440 texts; the flags re-sum to the printed accuracies exactly (0.4810 against the printed 0.481).
What would close it. The 2,000 generations per run, as the card’s re-run released its flags.
The 7.9% baseline is a looping regime that the base model’s own card warns about. Under the post’s settings (temperature 0.6, 32,768 tokens) 93.0% of the unedited MATH-500 answers hit the token limit (their merged.json). Qwen’s model card for Qwen3.5-0.8B, pinned at the same revision, says: “In thinking mode, we have observed that when using the recommended sampling parameters, Qwen3.5-0.8B is more prone to entering thinking loops compared to other Qwen3.5 models”; its recommended thinking-mode text settings are temperature 1.0 with 81,920 output tokens for competition math, and its own table reports IFEval 44.0 (thinking) / 52.1 (non-thinking) for the base model against the post’s 21.0. The multipliers in the post (“6×”, “3×”) are measured against that baseline. The edited IFEval 62.3 is above the card’s numbers, so the edit does something real; how much of 7.9 → 48.1 a no-edit configuration recovers was decided here on 2026-11-03 by one GPU run (§4), pre-registered the day before.
Closed 2026-11-03. Under the card’s settings the unedited model scores C1 7.2%, C2 22.3%, C3 44.0% (all samples, REFUSED counted wrong; the decided-subset accuracies and the first-sample ones are in the table). The pre-registered reading: mostly a settings effect.
48.1 is one run of several. The logs’ results table lists the three prompt-v3 runs the post reports (10, 25, 50 passes: 12.3, 14.1, 48.1 on MATH-500), two prompt-v4 runs (7.9 with no edit submitted; 14.1), earlier runs that submitted no edit, and prompt-v5 runs still pending. No variance across runs of the same budget is reported. The same table records the follow-up on Gemma 3 270M, a model that does not loop (8.5% of answers at the limit): fifty passes moved MATH-500 from 6.8 to 7.0 under lenient grading, within noise. Read together with §3: where there is no loop to break, the method found nothing to gain.
| claim | what the bytes say | verdict | |
|---|---|---|---|
| W1 | the same tensors in both checkpoints (names, shapes, dtypes) | 488 tensors each; no name, shape or dtype differs | CERTIFIED |
| W2 | exactly two tensors differ: layers 15 and 19 mlp.down_proj | layers.15.mlp.down_proj.weight, layers.19.mlp.down_proj.weight; every other tensor byte-identical | CERTIFIED |
| W3 | “131,072 weights” as the entries whose bits differ | 130,188 differ (65,260 + 64,928); their manifest’s 130,188 CERTIFIED | REFUTED |
| W3′ | 131,072 as the entries of the 64 addressed columns | 64 × 1,024 × 2 = 131,072 | CERTIFIED |
| W4 | “0.016%” over the release / the language model / its non-embedding weights | 0.015% / 0.017% / 0.026% | REFUTED |
| W4′ | “0.016%” over a round 0.8 B | 0.016% | CERTIFIED |
| W5 | the changed entries lie in exactly 64 columns per tensor, the ones the run’s manifest names | 64 and 64 columns; 1,024 and 1,023 of 1,024 rows | CERTIFIED |
| W6 | one direction: a witness (u, α) with bf16(W − u·α_c) = W′ on every changed column | the best witness found reproduces all but 42 and 65 of 65,536 entries per tensor (each within 6.3e-5 in absolute value, at entries where the subtraction nearly cancels); a failed search refutes nothing | REFUSED |
| W7 | the printed sha256 of both edited tensors (the card) and both base tensors (the manifest) | all four equal the bytes | CERTIFIED |
The sub-claims were written to corpus/wsp/claims.json before either decider ran. The Python decider (standard library: struct, mmap, array, Fraction) and the JavaScript one (Buffer, Uint16Array, Math.fround, BigInt) read the safetensors headers and the raw bytes themselves and agree on every count, column, hash and verdict. Two red controls must come out wrong and do: one bit flipped in a third tensor (W2 REFUTED with three tensors differing) and the layer-15 delta doubled (W6 REFUSED (the witness fails) and W3/W7 REFUTED on layer 15).
W6, in words. Astra’s own float32 submission was rank-1 to float32 precision (its manifest reports a directional residual of 2.5·10⁻⁷); the released checkpoint is that submission rounded to bfloat16, which adds a relative noise of about 2⁻⁹ to every entry. An SVD of the delta recovers the direction to that noise, and an interval search (every edited entry fixes the binary32 interval that rounds to it) brings the witness to within 42 + 65 entries of exact. The claim is CERTIFIED only with a witness that reproduces every entry; none was found, so it is REFUSED — which is a limit of the search, stated as such, not a finding against the edit.
The grader here (instruments/wsp/grade.py) reads the last closed \boxed{} itself, canonicalises both the prediction and MATH-500’s answer to an exact object, and says CORRECT, WRONG or REFUSED. It refuses what it will not canonicalise — an expression, an interval, a unit, a percentage against a bare number, a base-n literal against a bare number, the same elements in another order — rather than guess; 447 of the 500 gold answers are in its reach. The claimants graded with math-verify 0.9.0 on the same box. A second grader, written in JavaScript from the same convention and sharing no code with the first (instruments/wsp/grade.js), re-grades every pair both records graded: 7,440 pairs, 0 disagreements. Its first run, 3 November, found two: the answers 3i and −3i fell to text in the Python grader and were REFUSED against a complex gold, where the convention says WRONG; fixed the same day, both records re-derived, the floors of §4 unchanged.
| run (the post’s logs) | MATH-500, printed | the 2,000 flags re-summed | first samples ending in a box | of those, correct (theirs) | of those, correct (exact) | decided answers: agreement with theirs |
|---|---|---|---|---|---|---|
| unedited | 0.079 | 0.0790 | 35 of 500 | 35 | 35 of 35 decided | 500 of 500; 0 refused |
| edited, 50 passes | 0.481 | 0.4810 | 303 of 500 | 241 | 230 of 283 decided | 480 of 480; 20 refused |
Read exactly. The unedited model, when it finishes, is right: 35 first samples end in a box and all 35 are correct by both graders. The edited model finishes 303 times and is right 241 of them (79.5%). So the edit does not make the model better at the problems it already finishes; it makes it finish — which is exactly the mechanism the authors describe (a “loop direction” written by two MLPs) and exactly why a baseline that does not loop (§1, G5, G6) is the number the “6×” needs.
The card’s independent re-run. akhaliq’s re-run of the edited checkpoint prints 0.51; its 2,000 flags re-sum to 0.510. Its 20 released texts re-extract and re-grade as flagged (19 decided, 19 agree); the 25-problem base smoke test likewise (15 decided, all agree).
Extraction. On all 400 released texts of the post’s runs, the last box read here equals the answer string their pipeline extracted. The 20 refused edited answers are listed in the record with the reason each (a unit after the number, an equation for a number, a matrix, an interval, two multiple-choice letters, an order).
Pre-registered on 2026-11-02 (notes/wsp-stage3-preregistration-2026-11-02.md) before anything ran; run on 2026-11-03 on one rented NVIDIA H200 with vLLM 0.33.0 (the version of the card’s own re-run) from the pinned checkpoints and MATH-500, the prompt and the chat template of the post’s own evaluate.py; 3 conditions, four samples per problem as pre-registered; every generated text brought to the desk and graded by the grader of §3, 447 of the 500 gold answers in its reach. Accuracy is given two ways, as pre-registered: on the decided subset (CORRECT over CORRECT + WRONG) and as a floor (REFUSED counted wrong). The record (certs/wsp-baseline.json) holds every verdict and every pin.
| condition | model | first samples answered | first sample: decided / floor | all samples: decided / floor | hit the token limit | mean tokens | |
|---|---|---|---|---|---|---|---|
| C1 | the post’s settings (the control) | unedited | 8.2% | 7.8% / 7.8% | 7.2% / 7.2% | 93.1% | 30,866 |
| C2 | the card’s thinking-mode text settings | unedited | 24.4% | 22.3% / 22.0% | 22.5% / 22.3% | 75.0% | 67,765 |
| C3 | the card’s non-thinking text settings | unedited | 98.0% | 51.7% / 45.6% | 49.7% / 44.0% | 0.3% | 9,057 |
| the post, unedited, as printed (its settings are C1’s) | unedited | — | — | 0.079 (printed, 4 samples) | 0.93 | 30868.4345 | |
| the post, edited (50 passes), as printed | edited | — | — | 0.481 (printed, 4 samples) | 0.3305 | 22037.1475 |
The settings. C1: temperature 0.6, top_p 0.95, presence penalty 1.5, 32,768 tokens, thinking on. C2: temperature 1.0, top_p 0.95, presence penalty 1.5, 81,920 tokens, thinking on. C3: temperature 1.0, top_p 1.0, presence penalty 2.0, 32,768 tokens, thinking off. Each arm: the same prompt and chat template, seed 0, 4 samples per problem unless stated.
The pre-registered reading, applied as written. Stop rule A: C1 within 7.9 ± 3 (all-samples floor), else no verdict on the multiplier. Bands on the best no-edit card-settings arm (C2 or C3; the larger of the first-sample and the all-samples floor): below 15% HOLDS; 15 to 40% OVERSTATED; 40% or above MOSTLY A SETTINGS EFFECT; within 2 points of an edge: at the edge. The critic's 47% (non-thinking, no edit): REFUTED as stated unless C3 is within 5 points of 47. The multiplier re-stated as 48.1 over the best no-edit arm. Here: mostly a settings effect — C1 7.2%, C2 22.3%, C3 44.0%.
Read beside §3. The control C1 is the post’s own protocol run again on the unedited model; the card’s settings are what Qwen tells users to run. Whatever the table says, the counting is the same exact grader on every row, and a REFUSED answer is listed in the record with its reason, never scored.
What is not claimed. That the edit is small or large in effect beyond the released flags; that the mechanism is as described (read from the run’s notes, not probed here); anything about GSM8K, IFEval or the other benchmarks beyond re-reading their printed numbers; that math-verify is wrong anywhere (it agreed with the exact grader on every decided answer).
A viral thread answered the post with claims about the other experiments and about how the math edit was found. The ones that rest on released bytes are decided here from the logs the authors published (RikiyaT/lm_understand-logs @ 4d09ba6d; 190 files pinned; nothing of the claimants’ run, their .npz parsed with the standard library). A claim with no released source is REFUSED as a limit, not as a finding against whoever made it. The critic’s one number about the math — 47% with no edit in non-thinking mode — is a question for §4, not for the logs.
| claim | verdict | what the logs say | |
|---|---|---|---|
| M2 | the dose was picked over thousands of test answers (strengths tried: 1, 2, 3, 4, 6) | REFUTED REFUTED | 'thousands of answers': yes — 3680 generated answers were scored in counted passes before the dose was frozen at alpha 6 (turn t105, after pass 42), 2304 more in the frozen confirmation; 'test answers': no — every one of the 1504 problem ids in the 34 generate requests is a practice id (gsm8k_train 736, math_train 768), none is a MATH-500 problem by text or unique_id, no teacher-forced text contains a MATH-500 problem, and the practice file names the benchmarks it was decontaminated against. Limit: GSM8K test is not on disk here, so its text check rests on the claimants' released rule, not ours. |
| M3 | Frogger answered, Freeway credited (exp1_which_game) | CERTIFIED CERTIFIED | network 0bbcc3ea (truth freeway): Astra's Game line is 'Frogger — a simplified grid-based traffic-crossing variant.'; eval/scores.json maps it to guess 'freeway' and marks correct; the README states the keyword rule ('Frogger ... counts as Freeway') and the 4/4 counts it. The other three networks were answered by name (Breakout, Pong, Pac-Man). |
| M4a | car policy weights set to the hundreds (up to 876) | CERTIFIED CERTIFIED | max |value| over all actor tensors: 2.370 before, 876 after (actor0.b[125] = -876, from the quoted edit statement); 13 entries at or above 100 after the edit, 0 before; 6073 of 171664 entries differ from the network as shown (the claimants' 6,073 reproduced). |
| M4b | '100%' on one level / one run replayed | CERTIFIED REFUTED CERTIFIED (one level) · REFUTED (one run replayed) | one level: the training and evaluation name a single Kinetix level, l/h2_car_ramp; one replayed run: no — the 100% is 1,024 test episodes with sampled actions (success 1.0, mean length 73 steps, 74752 env steps) plus 256 search episodes (success 1.0) and the most-likely-action episode; a list of episode seeds is not released, so how many distinct initial states those episodes cover is REFUSED. |
| M5a | 20x over PPO; PPO weak; Astra lost in two other games | CERTIFIED CERTIFIED REFUSED CERTIFIED (20x over PPO) · CERTIFIED (lost in two other games) · REFUSED (PPO weak: a judgement, no released criterion) | exp5 (1,000-episode means): Breakout 517.4 vs PPO 25.3 (500M) / 15.8 (100M); Seaquest 53.1 vs 88.9 / 5.6; Asterix 53.2 vs 55.2 / 25.8 — the 20x and the two losses are both in the released table, the two losses against the 500M-step PPO only. exp6 Go: budget-0 network loses all three head-to-heads with the same-size PPO (42.6 / 26.3 / 36.3 %%), budget-50 wins them (74.2 / 70.0 / 78.5 %%). exp4: the PPO car network has 0 successes at every evaluation. 'Weak' has no released criterion; the PPO settings are released for a reader to judge. Nothing was re-run. |
| M5b | sycophancy 'failed from weights alone' | CERTIFIED CERTIFIED | the released score.json of attempts 1-3 has every answer 'A' at P(correct) = 0.5 (1/3 'correct' per attempt = chance, the README's own words); attempt 4 was stopped with one pair finished; the README says 'In four attempts with the weights alone, Astra could not tell' and reports the attention-tool round (3/3) as the result. Code, dataset, splits, eval scripts, runs and 18 sha256 lines are released; the 17 GB of models are not. |
Read exactly. The dose was chosen over thousands of answers — on practice problems from the GSM8K and MATH training splits, never on a MATH-500 problem, by id or by text (M2): the number the critic implies, tuning on the test set, is not in the logs. The game-identification scoring did credit “Frogger” as “Freeway” under a keyword rule the README states (M3). The car policy was edited to weights in the hundreds and evaluated on one level over 1,280 episodes, not one replay (M4). The PPO comparisons and the two lost games are in the released tables as stated; “weak” has no criterion to decide (M5a). The sycophancy round failed from the weights alone, in the authors’ own words (M5b).
Re-run: node tools/wsp-weights.js (both deciders, the controls, the pins; needs the cached inputs) · python3 -I tools/wsp-regrade.py · node tools/build-report-wsp.js.