The point of the residual audit is to surface places where the model is leaking. Most of the flagged trends turned out to be ARTIFACT- leakage from the y_cgs offset bias and went away when we recalibrated. Two are REAL-STRUCTURAL signals about the football landscape and got their own essays. This piece covers the third category, which has no good name: the audit found exactly what the audit was supposed to find.
1. Never-played picks cluster at the bottom of the residual
Pre-Phase-1.5, the diagnostics flagged rz_cgs × never_played with η² = 0.08, F = 935. Picks who never appeared in an NFL game had systematically low career-starts residuals. The pattern was enormous, flagged at p < 1e-200, and looked like the model needed fixing.
Post-calibration, the η² shrunk to 0.010: still significant, but eight times smaller. Most of the original effect was the global offset bias making negative residuals look more negative. What survives is the design intent: career_games_started_on_drafting_team is supposed to penalize never-played picks. A GM who took a sixth-rounder who never put on a uniform should post a negative residual on that pick, because the position-and-pick baseline expects some games started, and zero is below that. That's not a leak. That's the signal we ranked GMs on.
The methodological lesson: when a diagnostic flags a trend that is definitionally what the outcome was meant to capture, the right move is not to add a covariate. Adding never_played as a baseline regressor would zero out a chunk of the y_cgs signal; it would tell the model "ignore the gap between picks who played zero games and picks who started for a decade." That gap is the entire output we're after.
2. OFF/DEF/ST residuals differ slightly
The rz_cgs × pos_off_def cell came in at η² = 0.007, F = 37, p < 0.01. Career-starts residuals look slightly different by side of the ball. The natural worry: the position spline in the baseline GAM is misspecified at the OFF/DEF/ST level.
The breakdown looks like this:
| side | n picks | note |
|---|---|---|
| OFF | 5,339 | the bulk of the dataset; well-fit |
| DEF | 5,273 | also well-populated; matches OFF |
| ST | 217 | kickers, punters, long-snappers (sparse) |
With 217 special-teams picks across 45 years, the position-spline fit at the ST level is genuinely thin. Most of the η² in this cell is the ST tail: long-snappers and punters who, conditional on being drafted in the first place, beat or miss expectations idiosyncratically. We don't have enough ST volume to fit a stable per-position curve; the spline is mostly interpolating between OFF and DEF and the ST residuals are higher-variance.
The decision: leave it. Adding more granularity at the ST level requires more ST picks, which we don't have. The aggregate effect (η² = 0.007) is below the conventional small-effect threshold and the GM-level dispersion isn't dominated by the few ST picks any individual GM makes.
What's the point of the audit, then
Of the trends flagged at p < 0.01 in the original residual run, Phase 1.5 calibration resolved the largest as ARTIFACT-leakage. Two essays in this collection cover REAL-STRUCTURAL findings. This essay covers the cases where the audit's job was to confirm that the design intent was working: that y_cgs penalizes never-played picks and the ST class is sparse are statements about the model the audit verified, not bugs the audit found.
The framework matters because it's symmetric: an audit that only ever finds bugs would be making them up. An audit that only ever finds real-structural signals would be too credulous. Finding both, plus a third category of "yes, the model is doing what we asked it to do," is what an honest residual diagnostic looks like.