And the part that is not about HumanEval: mutation testing rates 75% of those items as fine. A suite that reliably kills sign-flips and constant-offsets can still accept a plausible off-by-one — so mutation-based sensitivity overstates suite quality.
Both ask: if the reference solution were wrong, would this item's own tests notice? The first asks it with mechanical mutants. The second asks it with realistic wrong answers. They do not agree, and the disagreement is the result.
Rows are the mechanical verdict — whether the item killed a majority of first-order mutants. Columns are what happened when a realistic wrong answer was tried on it.
18 of the 24 false passes sit in the top-right cell. Those items got a clean bill of health from mutation testing and then accepted a proven-wrong solution anyway. That is a claim about mutation testing as a method, not only about HumanEval — and it is the one result here that was not predictable in advance.
Not merely “sensitive” — these 10 items killed every single mechanical mutant thrown at them. Under mutation testing they are the best-behaved items in the benchmark. Each then accepted a solution an independent witness proves is wrong.
HumanEval/7 · HumanEval/9 · HumanEval/13 · HumanEval/19 · HumanEval/21 · HumanEval/48 · HumanEval/53 · HumanEval/58 · HumanEval/109 · HumanEval/152
Not selected for drama — these are the first three by item number. Each bug is the kind a model actually produces, not a mutation operator's idea of one.
| item | the bug | witness | correct | returned |
|---|---|---|---|---|
| HumanEval/0 | uses <= instead of strict <, so a distance exactly equal to the threshold is wrongly reported as close | has_close_elements([1.0, 2.0], 1.0) | False | True |
| HumanEval/7 | matches case-insensitively, so strings that only contain the substring in a different case are wrongly kept | filter_by_substring(['ABC'], 'a') | [] | ['ABC'] |
| HumanEval/9 | seeds the running maximum with 0 instead of the first element, so all-negative inputs report 0 | rolling_max([-1, -2]) | [-1, -1] | [0, 0] |
The full result, not a summary of it. Kill rate is the fraction
of mechanical mutants the item's suite killed — note how many 1.00 rows sit in the FALSE-PASS
filter. A perfect mutation score is not evidence the suite is sound.
| item | outcome | mechanical | kill rate | the bug | witness — correct vs returned |
|---|
Candidates were written by a model asked to introduce a subtle bug. They are realistic but not a random sample of real model errors, so 14.7% is a rate for this adversary, not a universal constant.
One candidate per item. A different bug on the same item might well be caught. The witness proves wrongness at a single input — sufficient for “wrong”, and silent on how wrong.
The authors were told not to look for the test suite. Targeting the tests instead of writing a realistic bug would invert the measurement. That was an instruction, not an enforced sandbox.
HumanEval being weak is already known. EvalPlus (arXiv:2305.01210) established it and responded by adding ~80× more tests. This is a different measurement — per-item acceptance of wrong solutions, not added inputs — and a different response: withhold the result rather than strengthen the suite. The numbers are not comparable and are not presented as if they were.
The 10 OUT-OF-SCOPE items are the mutation prober's limit, not a benchmark finding: one-liners with no comparison, arithmetic or literal for the operator set to reach. They are still measurable by witness, which is why they appear in the cross-tab.
No claim is made about any model's capability. Nothing here ranks anything.
The decisive assertion runs the identical wrong candidate with the identical witness against a thorough and a thin suite, and requires the verdicts to differ. A scorer that returned the same verdict for both would be measuring the candidate rather than the suite, which is the one thing it must never do.
gold control the unmutated reference must PASS — 0 of 164 failed null control `return None` must be KILLED — else the item is VACUOUS compile check every emitted mutant parses — else it measures the prober discard branch a secretly-correct candidate can never score FALSE-PASS no silent zero an item with no mutants reports null, never 0.0
Written after the headline, and recorded as such. This unit produced 14.7%
before its false-pass scorer had a battery of its own — the exact ordering the unit argues against. The
battery (tests/test_falsepass.py, 7 checks) exists now and the sequence is not tidied away.