The Forecast Gym: the test set that cannot leak

Evaluate a model's forecasting on past data and you are asking a question the training corpus may already answer. Cutoff dates are self-reported, contamination audits are heuristic, and a re-scraped benchmark leaks the moment it is published. No split protocol can prove a negative about what a model has seen.

The Gym inverts the burden. A forecast is committed — content-addressed by sha256, timestamped — while the outcome is still in the future, so "the model saw the answer" is not improbable but impossible: the answer did not exist. Hygiene is then a property a stranger can check mechanically: the ledger is append-only in a public git history, so every commit carries two independent timestamps (the row's, and git's) that must both precede the target day. The same discipline that removes the answer key from the matmul eval — there the reference is a proof, here the reference is the future.

A commit row pins {id, target, madeAt, targetTime, sha256(forecast)} with madeAt < targetTime enforced — a backdated commit is refused, a premature score is refused, a second score of the same id is refused, and a sealed commit whose revealed payload does not hash to its committed sha is refused as tampered. Each refusal is a red control in the battery: the mechanism is proved able to say no before it is trusted to say yes.

Sealed commits are how competing proposers enter without reading each other: the ledger shows only the sha until scoring, when the payload is revealed and verified against it. A model campaign commits sealed; the reveal is part of the scoring transcript.

Every scored forecast pays the Winkler interval score at its own committed miss-rate α: width + (2/α) · distance-outside, computed in exact rationals and stored as a string. The score is strictly proper for central intervals — its expectation is uniquely minimized by reporting the true quantiles — so there is no strategy against it: hedge wide and the width term pays, claim narrow and the miss term pays. From the battery's hand-computed row: forecast [10, 20] at α = 1/4 scores 10 when the outcome is 15, and 10 + 8·6 = 58 when the outcome is 26.

This is the probabilistic sibling of certify(): there, a claim is decided and the reward cannot pay a false claim; here, a forecast is scored and the reward cannot pay a dishonest one better than an honest one — in expectation, which is the strongest thing a probabilistic channel can promise. Both channels share the discipline: append-only ledgers, red controls first, exact arithmetic everywhere a number is decided.

Doctrine: prediction enters as a proposer, never an authority, and a proposer that misses its certified coverage stops being admitted until recalibrated. The Gym computes that rule instead of judging it. A proposer claiming coverage p with k of m scored forecasts covered is held to the exact binomial tail P[X ≤ k], X ~ Binomial(m, p) — BigInt rationals, no float — and is DEADMITTED when the tail falls to 1/20 or below. The verdict reads: "if the claimed coverage were true, a record this bad or worse has this exact probability." The tail is the certificate; the bar is a stated convention; only under-coverage prunes, because over-coverage is the conservative direction and the score already prices it.

A deadmitted proposer's new commits are refused by the gym runner, and its deadmission stands in the ledger permanently — wrong forecasts are exhibits here, not embarrassments. The forced dumb baseline below exists to be pruned in public: the mechanism firing on schedule is the demonstration.

A proposer is anything that emits interval forecasts with a claimed coverage for the named future targets. Submissions are sealed commits: the sha enters the public ledger before the target day, the payload is revealed and hash-verified at scoring, attribution required. Scoring and admission are deterministic, so a submission cannot be argued with — only outperformed.

node tools/forecast-gym.js record prints the board; node instruments/forecast/battery.js proves the machinery on your machine first, reds included. Model campaigns through the same path are spend and run on the operator's word, like every campaign on the eval shelf.