The Edgework model · honesty surface

How Calibrated Is It?

A forecast is only as good as its calibration: when it says 70%, does it happen 70% of the time? Here is the reliability curve from a held-out backtest — shown straight, including where the model is over-confident.

320

Calls (team × snapshot)

0.4372

Log-loss

vs 0.693 coin-flip

0.1484

Brier

vs 0.25 coin-flip

Dashed = perfect calibration. Point size ∝ sample. Orange = over-confident.

Over-confident band: 0.5–0.8

In this range the model says “likely” but reality is closer to a toss-up — its discrimination is modest, so its confident-middle calls are softer than stated. We show this rather than smooth it over.

FC-C.1 snapshot backtest (n=320). Well-calibrated at the extremes; MILD OVER-CONFIDENCE in the 0.6-0.8 band — disclosed, recal deferred until 2026-27 makes it 3 seasons.

Predicted	n	Observed	Gap
0.0–0.1	60	0.067	-0.034
0.1–0.2	23	0.261	-0.114
0.2–0.3	37	0.297	-0.048
0.3–0.4	24	0.417	-0.070
0.4–0.5	23	0.435	+0.014
0.5–0.6	16	0.438	+0.107
0.6–0.7	21	0.476	+0.179
0.7–0.8	19	0.474	+0.270
0.8–0.9	33	0.879	-0.033
0.9–1.0	64	1.000	-0.028