An evaluation graded against reference values inherits the failure class of whatever computed them — and for mathematical ground truth the computing pipeline is usually floating point, checked by digit agreement. This note holds three certified specimens of answer keys going wrong in ways reruns and digit cross-checks provably cannot catch, and one working design that removes the answer key altogether. Every specimen was re-proved during the build that produced this page.
The reference values in mathematical answer keys are computed — by scripts, by CAS calls, by a model run once and trusted. When the pipeline is floating point, its silent failures become the key's silent failures, and the usual defenses do nothing: a RERUN of the same pipeline reproduces the same artifact bit for bit, and a DIGIT CROSS-CHECK against an independently computed float value agrees, because both pipelines drop the same mass the same way. The failure class lives inside the key, and a model that reproduces the artifact grades as correct — while a model that computes the true value grades as wrong.
This is not hypothetical. Each section below is a certified specimen from this site's audits — re-proved, not cited, during this page's build.
A GPT-published constant on Erdős #852 — an Euler product over primes — is wrong from its 12th significant digit. The wrong digits are not near the truth by accident: re-run live during this build, the naive double-precision product over 148,932 primes emits 0.0752403861777419 — the published value, digit for digit — because 87.5% of its factors round to exactly 1.0 and silently vanish. The exact partial product, a strict lower bound with no tail estimate needed, already exceeds the printed value: REFUTED, re-decided this build. The published constant is the bug, printed.
If that number sat in an answer key, every float-faithful model would score correct and every exact model would score wrong. The full mechanism, the certified correction, and the failure taxonomy are in the Erdős #852 report; this is its §6 argument, promoted to a page of its own.
Validating a key by matching digits assumes agreement implies identity. The impostor catalog is the certified counterexample set: 21 published constants that agree with simple closed forms — and are provably not equal to them, each refutation one exact BigInt comparison at the full published precision. The deepest specimen (A271880, re-derived this build) agrees with a plain rational for 62 significant digits before exact arithmetic separates them. Any digit-matched validation shallower than that certifies the impostor; twenty digits — a "discovery" threshold elsewhere — is not close. Agreement depth is evidence about your precision budget, never about identity.
The subtlest class: the key is wrong even though the computation behind it was right. The Ramanujan Machine's 2022 mixed-zeta sheet prints an identity that is FALSE as printed — a sign slip in the constant term, one of three typographic errors on the sheet — while the continued fraction its own polynomials define converges to the corrected value exactly: both directions re-certified on one enclosure during this build. No rerun of the discovery pipeline catches this, because the pipeline was never wrong; the print is. A key transcribed from a paper inherits the paper's typos as ground truth, and a model reproducing the (correct) computation grades as wrong against the (false) print.
The structural fix is to grade tasks where no reference value exists to contaminate: ask the model to EXHIBIT a witness — a decomposition, a certificate, a construction — and let the grader re-derive the claim from the witness alone, exactly. The matmul eval on this site is the running instance: 268 frontier-model proposals graded to date, every certified row a theorem, every refuted row a proof of error, zero false certifications — measured, and structurally guaranteed, because there is no answer key to be wrong.
Three rules transfer to any evaluation built on computed ground truth. Screens may only PRUNE: no float comparison ever admits a claim. Grade with certificates, not tolerances: the verdict should be checkable by re-derivation, not by proximity to a stored number. And put red controls in the grader: deliberately false submissions that must fail, run before anything real is graded — a grader that has never rejected a forgery is not known to reject forgeries. The methods note states the discipline in full; every gate on this site runs under it.