Every auto-scored benchmark rests on a verifier, and the verifier is almost never audited — because auditing it looks circular. It isn't, if you separate the directions. Measured across 164 verifiers: 10 of the 103 that rejected every mechanically-broken variant — a perfect score — still accepted a provably wrong answer. That is 10%, and it is invisible to the checks normally used.
To ask “is this verifier right?” the obvious move is to run it — which is the thing in question. That circle is why verifiers go unaudited, and it is escapable: a verifier fails in two independent directions, and only one of them is circular to test.
Rejecting a correct answer is cheap to test — run a known-good solution. Accepting a wrong answer is the hard one, because you must establish the answer is wrong without asking the verifier. Here that is a witness: one concrete input where the candidate and the reference solution disagree, evaluated against both. If they agree, the candidate is not shown to be wrong and is discarded — counted in no denominator. Without that discard, a secretly-correct candidate slipping past a weak verifier would be recorded as a false positive, manufacturing the exact result the audit exists to test for.
Of the 103 verifiers that rejected every single mechanically-broken variant thrown at them, 10 then accepted a solution a witness proves is wrong — 9.7%, 95% CI [5.4, 17.0].
An earlier version of this page said “D2 does not predict D3”. That was measured at one point and overstated. Binned across the whole range the relationship is real, weak, and in the expected direction — correlation -0.166 over n=153. Corrected here rather than quietly softened.
| D2 kill rate | n | failed D3 | rate | 95% CI | |
|---|---|---|---|---|---|
| 0.00–0.50 | 3 | 1 | 33.3% | [6.2, 79.2] | |
| 0.50–0.75 | 19 | 4 | 21.1% | [8.5, 43.3] | |
| 0.75–0.90 | 23 | 3 | 13.0% | [4.5, 32.1] | |
| 0.90–1.00 | 5 | 1 | 20.0% | [3.6, 62.5] | |
| 1.00 (perfect) | 103 | 10 | 9.7% | [5.4, 17.0] |
| verdict | n | meaning |
|---|---|---|
| ADEQUATE | 137 | all three directions hold |
| FALSE-POSITIVE | 24 | accepted an answer proven wrong — the expensive failure |
| WEAK | 2 | fails even the cheap direction |
| UNAUDITED-D3 | 1 | D1/D2 green and D3 never asked — the blind spot, never scored as adequate |
| BROKEN | 0 | rejects its own reference solution |
UNAUDITED-D3 is a verdict, not a pass. A verifier nobody has tried to fool is not thereby sound, and recording it as adequate would be the vacuous green this whole audit exists to prevent.
The whole list, not a summary of it. Read the D2 column: these are the verifiers that failed the expensive direction, and many of them scored a perfect 1.00 on the cheap one. The witness is the input that proves the accepted answer was wrong.
| item | D2 | the bug it accepted | witness |
|---|---|---|---|
| HumanEval/0 | 0.80 | uses <= instead of strict <, so a distance exactly equal to the threshold is wrongly reported as close | has_close_elements([1.0, 2.0], 1.0) |
| HumanEval/7 | 1.00 | matches case-insensitively, so strings that only contain the substring in a different case are wrongly kept | filter_by_substring(['ABC'], 'a') |
| HumanEval/9 | 1.00 | seeds the running maximum with 0 instead of the first element, so all-negative inputs report 0 | rolling_max([-1, -2]) |
| HumanEval/13 | 1.00 | the divisor search range excludes min(a, b) itself, so when one number divides the other the answer is too small | greatest_common_divisor(4, 8) |
| HumanEval/19 | 1.00 | 'six' and 'seven' are transposed in the word list, so they sort into each other's positions | sort_numbers('six seven') |
| HumanEval/20 | 0.60 | uses <= when updating the best pair, so ties are broken toward the last equally-close pair instead of the first | find_closest_elements([1.0, 2.0, 3.0, 4.0]) |
| HumanEval/21 | 1.00 | rounds each rescaled value to two decimals, losing precision on values that are not exact hundredths | rescale_to_unit([1.0, 2.0, 4.0]) |
| HumanEval/22 | — | uses type(v) is int instead of isinstance, so booleans (which are ints) are dropped | filter_integers([True]) |
| HumanEval/23 | — | strips surrounding whitespace before counting, so leading and trailing spaces are not counted | strlen(' a ') |
| HumanEval/28 | — | strips whitespace from each string before joining, altering the concatenation | concatenate([' a ', 'b']) |
| HumanEval/29 | — | tests for the prefix anywhere in the string instead of only at position 0 | filter_by_prefix(['bca'], 'a') |
| HumanEval/31 | 0.80 | the trial-division range stops before int(sqrt(n)), so squares of odd primes are reported prime | is_prime(9) |
| HumanEval/35 | 0.00 | initialises the maximum to 0 instead of the first element, so an all-negative list returns 0 | max_element([-5, -2]) |
| HumanEval/40 | 0.67 | the middle loop starts at i rather than i + 1, so one element can be used twice in the triple | triples_sum_to_zero([1, -2]) |
| HumanEval/48 | 1.00 | lowercases the text first, so it treats case-differing strings as palindromes | is_palindrome('Aba') |
| HumanEval/53 | 1.00 | increments x by one y times, so a negative y produces an empty range and x is returned unchanged | add(2, -3) |
| HumanEval/58 | 1.00 | never de-duplicates, so an element repeated in l1 appears multiple times | common([1, 1, 2], [1, 2]) |
| HumanEval/59 | 0.54 | the trial-division bound is strict (factor*factor < n), so a perfect square of a prime is never divided and the square itself is returned | largest_prime_factor(4) |
| HumanEval/86 | — | uses split() rather than split(' '), collapsing runs of blank spaces | anti_shuffle('a b') |
| HumanEval/88 | 0.86 | the parity test is taken on the already-sorted copy, so it uses min+max instead of the original first+last elements | sort_array([3, 1, 2]) |
| HumanEval/95 | 0.91 | uses k == k.lower() instead of k.islower(), so caseless keys such as '123' are treated as lower case | check_dict_case({'123': 'a'}) |
| HumanEval/109 | 1.00 | only linear adjacent pairs are checked; the wrap-around pair (last vs first) is omitted, so a single descent is accepted even when the array is not a rotation of the sorted array | move_one_ball([2, 3, 1, 4]) |
| HumanEval/124 | 0.58 | February is capped at 28 days instead of the 29 the spec requires | valid_date('02-29-2024') |
| HumanEval/152 | 1.00 | returns the signed difference instead of the absolute difference | compare([1], [3]) |
Roughly a day's work for a benchmark you already own, and the expensive step is step 3 — producing plausible wrong answers is where the cost sits, which is precisely why almost nobody does it and why D2 gets used as a proxy for it.
Everything above rests on ONE plausible-wrong candidate per verifier, so it measures what a single attacker happened to find. Three independent adversaries were then run against the same cohort — the ones that scored a perfect D2 — each blind to the tests and to the reference solution, because the files they were given contain only the problem statement. The blind is a property of the input, not an instruction anyone has to be trusted to follow.
95% CI [7.5, 20.4]. And the shape matters more than the number:
breached by all 3 adversaries : 3 breached by exactly 2 : 5 breached by exactly 1 : 5 <-- each of these was rated ADEQUATE by the other two never breached : 90
Identical items, identical verifiers, one attempt each. This is the second result the experiment buys: unlike accuracy on HumanEval — where frontier tiers sit at 99–100% and no ranking can move — writing a wrong answer that survives a verifier has plenty of headroom.
| adversary | provably wrong | got past | attack rate | 95% CI | discarded |
|---|---|---|---|---|---|
| prior-unlabelled | 103 | 10 | 9.7% | [5.4, 17.0] | 0 |
| sonnet | 103 | 10 | 9.7% | [5.4, 17.0] | 0 |
| haiku | 93 | 4 | 4.3% | [1.7, 10.5] | 7 |
The intervals overlap, so the tier gap is suggestive and not established. One sample per item, one prompt, no repeats — an attack rate here is a property of (model, prompt, single sample), not of a model in general. Do not read it as a capability ranking.
The cleaner signal is the last column. “Discarded” counts candidates whose witness failed to separate them from the reference — the adversary believed it had written a wrong answer and had not. That is a reliability failure rather than a weak attack, and it does not overlap: the smaller model produced several, the others none.
prior-unlabelled is the original run, included for the exposure union only. Its model was never recorded and its blind was an instruction rather than an enforced property of the input, so it is not a labelled point in any comparison between tiers.
Rates on this page are conditional on a perfect D2 score, since that is the cohort attacked. They are not whole-benchmark rates and must not be quoted as such.
Epoch AI runs FrontierMath: Open Problems — a benchmark of genuinely unsolved research mathematics, where each problem ships a custom verification program. It is a good place to look precisely because Epoch publishes its methodology and its retractions; most benchmarks disclose neither.
Their published answer to “How do you know the verifiers work correctly?”:
“We test the verifiers on variants of the open problems where solutions are known.”
FrontierMath: Open Problems FAQ · fetched at source 2026-08-08 · https://epoch.ai/frontiermath/open-problems/about/faq
Variants where solutions are known is D1. It establishes that the verifier accepts correct work. It does not, on its own, establish that the verifier rejects incorrect work, and the measurement above is that these two do not follow from one another.
They act on evidence in that direction. On 2026-07-31 two problems were retired because:
“we determined that the verifiers for these problems would not detect correct solutions with high enough fidelity.”
Open Problems changelog 2026-07-31 · fetched at source 2026-08-08 · https://epoch.ai/frontiermath/open-problems
That is the false-negative direction — a verifier that would not recognise a correct solution — found and acted on. The mirror direction, a verifier that accepts an incorrect solution, is the one this audit measures, and it is the more expensive of the two: a false negative costs a retired problem, while a false positive means a system is credited with solving an open problem in mathematics that it did not solve.
Epoch is explicit that a positive result is evidence rather than proof:
“We allow verifiers that provide strong numerical evidence that the AI system has solved the problem, without constituting full proof.”
FrontierMath: Open Problems FAQ · fetched at source 2026-08-08
No claim is made about the FrontierMath verifiers. They are not public — the FAQ states “Access to the verifiers is available for purchase by any party.” — and nothing here has been run against them. The measurement above is on HumanEval. Reading it as a result about Epoch would be exactly the unlicensed inference this page argues against.
The witness method does not transfer to open problems as built. It works by comparing a candidate against a known-correct answer. These problems are unsolved; there is no reference to compare against. It transfers cleanly to benchmarks where answers are known.
The red control does transfer, and needs no known answer. You do not have to detect wrongness — you can construct it: feed the verifier deliberately broken candidate solutions and require rejection. That is D2, it is cheap, and it is a lower bound rather than a proof of adequacy — which is the whole point of the 10% above.
D3 here rests on one adversarial candidate per item, written by a model asked to introduce a subtle bug. It is a lower bound on the false-positive rate for that adversary, not a universal constant: a stronger adversary would find more, a different one might find different items. One candidate, one witness, one input.
This is not a novelty claim. Mutation testing is decades old and EvalPlus (arXiv:2305.01210) already established HumanEval's suites are weak by adding ~80× more tests. What is offered here is narrow: the D2/D3 gap as a measured quantity, and the witness as the thing that makes D3 non-circular.
The audit takes a reference solution, a way to construct broken variants, an adversarial candidate with a witness, and the verifier under test. Supply those four and it audits a proof checker, an exact-answer comparator or a domain validator on the same three axes. HumanEval is the worked demonstration, not the scope.
D1 verifier(reference) must ACCEPT
D2 verifier(mechanically_broken) must REJECT <- cheap, automatic, a LOWER BOUND
D3 verifier(plausibly_wrong) must REJECT <- needs a witness; the one that matters
witness(candidate) != witness(reference) -> provably wrong, audit it
witness(candidate) == witness(reference) -> DISCARD, count it nowhere