cert-machine · report · generated from the records
Does an AI’s Lean proof prove the right statement? One command to check
A Lean proof can be correct and still prove the wrong thing. Lean’s kernels check the proof; this tool checks the statement, with exact tests instead of a reader’s opinion. Two AI-written refutations of Erdős problems, waiting on Google DeepMind’s list of open problems, went through it.
Lean’s kernels answer the first question. The second is the one this tool adds: the same statement, wrong ones refused, and the statement compared with the problem’s own words on small cases.Records: certs/lean-intake-erdos-488.json · sha256 3d55fa1fe80a; certs/lean-intake-erdos-786i.json · sha256 98066cbd66c4
claims checked
2
Erdős #488 and #786(i)
planted wrong statements refused
16 of 16
and 10 of 10 rewordings accepted
planted defects caught
7 of 7
a false step, a sorry, an axiom, native_decide
a fresh claim, rules to record
51 min
on a laptop
The mathematics is the claimants’ (Declan Gessel’s, with OpenAI’s GPT-6 Astra, for both refutations); the statements are Google DeepMind’s. This page checks their proofs against those statements. The tool is mine, built with Claude (Anthropic).
read this first
What it does not decide
Whether the formal statement is the problem. That stays a reading; the small cases are evidence, not proof.
“The same” means the same after unfolding definitions. Statements equivalent only by an argument are refused (all 3 planted ones were).
Small cases compare true and false. A change that never flips an answer there is invisible to them.
Some statements have no small cases. #786(i) is about densities of infinite sets: refused there, as planned.
Four of Lean 4.35’s bundled checkers were not run. Both claims are on Lean 4.33.1; leanchecker-paranoid, lean4lean, con-leche, con-ron come later.
does the check work?
Every planted test came out as it should
Each run tests the check itself. Wrong versions of the statement are planted (a dropped hypothesis, a changed constant, a shifted interval) and must be refused; rewordings of the same statement must be accepted; broken copies of the proof must be caught. One square per test; hover for what it was.
The planted tests of both runs. The three filled squares in the third row are the stated limit: statements that are logically the same but differ by more than unfolding a definition are refused.Records: certs/lean-intake-erdos-488.json · sha256 3d55fa1fe80a; certs/lean-intake-erdos-786i.json · sha256 98066cbd66c4The build refuses this figure if a planted statement or defect is not as expected.
Erdős #488
DeepMind’s statement against Erdős’s sentence
Erdős asked whether, for a finite set A and B the numbers with a divisor in A, |B ∩ [1,m]|/m < 2|B ∩ [1,n]|/n always holds for m > n ≥ max(A). Gessel’s proof refutes DeepMind’s Lean version of it; this run reached the same verdicts as a separate hand-built check of 6 November.
Then the new check: Lean evaluated DeepMind’s statement part by part on 3,024 small cases (the kernel confirmed every value), and a program written from Erdős’s sentence alone evaluated the same cases.
Every part agrees wherever Erdős’s sentence makes sense. The 20 cases where the whole statements differ all have A = {0}: B is empty there, the sentence reads 0 < 0 and is false as written. DeepMind’s statement adds the hypothesis 0 ∉ A for exactly this (their pull request #256); the check found the reason without being told.Records: certs/lean-intake-erdos-488.json · sha256 3d55fa1fe80a; instruments/intake/erdos-488.js · sha256 a2dc8058927bThe build refuses this figure if the cases stop adding up, or a disagreement is not one with A = {0}.
Erdős #786(i)
A claim the tool had not seen
Part (i) asks whether some set of density above 1 − ε has all its equal products of equal length. Gessel’s refutation (pull request #5299, open since 6 September) states its own versions of the two definitions it needs. The kernel accepts it against DeepMind’s declaration as it stands today, so the definitions match.
It also turned up one thing for the pull request: DeepMind reworded this statement on 14 October, so the change no longer applies and GitHub reports a conflict. The proof still refutes the current statement.
From the spec committed to the record, on a laptop. The first run found a defect in the tool (its test files missed one line of DeepMind’s file); it was fixed and the planted statements run again, with no statement or expectation changed.Records: certs/lean-intake-erdos-786i.json · sha256 98066cbd66c4; corpus/lean-intake/erdos-786i/runs/2026-11-09/_status.txt · sha256 16a5b9f23555The build refuses this figure if the clock falls out of order.
what I got wrong
Six corrections, made in the open
#488: a “wrong” statement was the same. Counting from 0 changes nothing, since 0 is never counted; the kernel saw it. My label was wrong (14 of 15 as first written).
#488: the small cases found more than I expected. The 20 cases above, all explained.
#488: one planted statement was invisible to the small cases. Counting from 2 changes counts, never answers: the limit, measured.
#786(i): a defect in the tool. Fixed and re-run; then 12 of 12 came out as written.
#12 (ii): a defect in the tool (10 November 2026). Lean v4.27.0 ships no leanchecker. The tool counted the step that could not start as a kernel that did not accept, and the verdict came out REFUSED. Fixed: a checker the toolchain does not ship is now listed as not run. No statement or expectation changed, and nothing was re-run.
The batch: 6 planted statements tested nothing (11 November 2026). In 5 of the nine claims below, a planted statement that swaps a number type did not compile, so the proof was never tried against it. My records counted each as a prediction met. The table now counts only statements that reached the check. No verdict changes. The records stay as written; the tool no longer counts this way, and instruments/intake/SPEC.md lists what else those records misstate (their cost, machine and checker lines).
Each spec was committed before its run, and each correction is dated beside the first outcome.
the batch
Nine claims, specs written by a program
From here the planted statements are no longer written by hand. claimbox writes each spec from the claim’s pins: rewordings that must be accepted, wrong statements by fixed rules (a hypothesis dropped, a strict inequality made weak, a numeral moved by one), and a prediction only where the kernel’s rules make it certain. Everything else the kernel decides. Each spec was committed before its run.
#488 ran again first, against its known answers, then two open pull requests: four of Matt Stokes’s Erdős proofs and four of Daniel Puri’s. Across the nine, the kernel refused 60 of 62 changed statements. The two it accepted, both on #488, count from 0 instead of 1, and 0 is never counted: the same statement, which is why such changes carry no prediction.
None of the other eight has small cases a program can read the original sentence on, so the comparison with the sentence ran only for #488. One claim needed a dated amendment: its statement uses a shorthand that DeepMind’s file keeps private, which the planted statements could not see. The first run, refused, is kept in the repository’s history.
files that repeat the statement
When the proof file restates the statement
Many proofs ship as a pair: a statement file and a solution file that declare the same names. Lean cannot load both at once, so the check above could not even be written for them. Since 10 November 2026 the tool has a mode for such files. It decides in three steps.
A copy of the statement file. One line is added (namespace IntakeCanon), so the copy loads beside the proof file. Without that line the copy is the original, byte for byte.
The copy is shown to be the original. Every declaration the check uses is compared with the original’s, constant by constant: types, the values of definitions, inductive types. Any difference refuses the claim.
Then the same check, against the copy. The kernel must accept the proof as a proof of the copy’s statement, so every definition the proof file restates has to unfold to the original one.
What it cannot do:
Restated structures. Lean treats a structure or an inductive type written out again as a new type, even word for word. Such a claim is refused.
Restated imports. If the proof file also repeats names from a file the statement imports, the two still collide.
Automatic names. An instance that Lean names after its namespace would have no counterpart in the copy. The tool refuses and says so.
Planted statements. The open lines a spec gives for them are written as given, inside the copy’s namespace; the tool does not adapt them.
Axioms the statement file declares. The proof may rest on its own axioms of the same names only if each matches the original, name for name. The verdict is then conditional on them and says so. Nothing here proves them.
Erdős #12 (ii)
A proof that restates DeepMind’s definition
The proof is AlphaProof Nexus’s (Google DeepMind), from its published results; the statement is DeepMind’s too, from its list of open problems. Erdős #12 is about infinite sets of whole numbers in which no element divides the sum of two different larger ones. Part (ii) asks whether some fixed c > 0 gives every such set fewer than N^(1 − c) elements up to N, for infinitely many N. The proof says no.
The proof file writes out DeepMind’s definition of such a set again, under DeepMind’s own name (Erdos12.IsGood), so the two files cannot be loaded together. The proof itself was already checked here on 20 October, by two kernels (the AlphaProof Nexus report). The question this time was the tool’s: does the new mode reach the same answer, and does it still refuse a wrong statement?
The copy is DeepMind’s file. Byte for byte without the added line, and the declaration the check uses is identical (2 constants compared).
The kernel accepts the proof against the statement. The two statements differ as written and agree once the restated definition is unfolded. Only the three standard axioms sit under the proof.
2 of 2 rewordings accepted, 4 of 4 wrong statements refused. The wrong ones: the answer flipped: answer(True) for answer(False); c > 0 weakened to c ≥ 0; the exponent 1 − c made 1 + c; the hypothesis IsGood A dropped.
2 of 2 planted defects caught. A sorry and an added axiom, each shown in the axiom list.
Lean and nanoda accept. leanchecker was not run: Lean v4.27.0 does not ship it. Nor were the four later checkers named at the top.
Small cases: refused, not run. No finite case decides a statement about every infinite set.
The verdict is CERTIFIED, from one run of 17 minutes on a laptop: the kernels named accept the proof as a proof of DeepMind’s statement, and the check still tells a wrong statement from the right one. DeepMind’s file declares no axioms, so nothing here is conditional. Whether that statement is Erdős’s problem stays a reading.