Model win probabilities against the de-vigged market price.
97 graded game-forecasts across 42 games. Lower Brier is better. The market column is the de-vigged closing moneyline, and the difference is paired on the same games.
| group | n | brier | log loss | straight-up | market brier | diff (model − market) | t | read |
|---|---|---|---|---|---|---|---|---|
| identified / blind / game / vv2 | 16 | 0.2271 | 0.6466 | 68.8% | 0.2166 | +0.0105 ± 0.0264 | +0.40 | market better, not significant |
| identified / blind / game / vv3 | 10 | 0.2057 | 0.5992 | 70.0% | 0.2540 | -0.0483 ± 0.0377 | -1.28 | model better, not significant |
| identified / blind / slate / vv2 | 15 | 0.2464 | 0.6852 | 60.0% | 0.2206 | +0.0259 ± 0.0340 | +0.76 | market better, not significant |
| identified / informed / game / vv2 | 32 | 0.2098 | 0.6126 | 71.9% | 0.2124 | -0.0026 ± 0.0060 | -0.44 | model better, not significant |
| identified / informed / game / vv3 | 24 | 0.2368 | 0.6640 | 58.3% | 0.2337 | +0.0032 ± 0.0083 | +0.38 | market better, not significant |
A |t| below 1.96 is noise. Do not tune on it.
| run | n | brier | market brier | diff |
|---|---|---|---|---|
| T1_open | 9 | 0.2205 | 0.2243 | -0.0038 |
| T2_informed | 16 | 0.2118 | 0.2166 | -0.0048 |
| T3_final | 31 | 0.2266 | 0.2232 | +0.0033 |
| bucket | n | predicted | actual | gap | |
|---|---|---|---|---|---|
| 0.1–0.2 | 1 | 17.0% | 0.0% | ▼ -17.0 pp | |
| 0.2–0.3 | 1 | 26.0% | 0.0% | ▼ -26.0 pp | |
| 0.3–0.4 | 2 | 37.5% | 50.0% | ▲ +12.5 pp | |
| 0.4–0.5 | 10 | 43.6% | 50.0% | ▲ +6.4 pp | |
| 0.5–0.6 | 7 | 57.6% | 71.4% | ▲ +13.9 pp | |
| 0.6–0.7 | 9 | 64.6% | 66.7% | ▲ +2.1 pp | |
| 0.7–0.8 | 10 | 73.6% | 60.0% | ▼ -13.6 pp | |
| 0.8–0.9 | 1 | 85.0% | 100.0% | ▲ +15.0 pp |
When it says 65%, does it happen 65% of the time? A model that never beats the market but is honestly calibrated is still useful; an overconfident one is actively harmful.
| game | wk | side | price | model | market | edge | kelly stake | result | flat P&L | cumulative | running ROI |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 2026_01_NYJ_TEN | 1 | TEN | -118 | 57.0% | 51.7% | ▲ +5.3 pp | 0.016 | lost | -1.00 | -1.00 | -100.0% |
| 2026_01_DAL_NYG | 1 | NYG | +140 | 47.0% | 40.0% | ▲ +7.0 pp | 0.023 | won | +1.40 | +0.40 | +20.0% |
| 2026_01_DEN_KC | 1 | DEN | +110 | 46.0% | 54.3% | ▼ -8.3 pp | 0.030 | lost | -1.00 | -0.60 | -20.0% |
| 2026_02_CLE_TB | 2 | CLE | +330 | 73.0% | 77.7% | ▼ -4.7 pp | 0.012 | won | +3.30 | +2.70 | +67.5% |
| 2026_02_JAX_DEN | 2 | JAX | +124 | 44.0% | 57.2% | ▼ -13.2 pp | 0.051 | lost | -1.00 | +1.70 | +34.0% |
| 2026_02_SEA_ARI | 2 | ARI | +180 | 43.0% | 34.3% | ▲ +8.7 pp | 0.028 | lost | -1.00 | +0.70 | +11.7% |
| 2026_02_WAS_DAL | 2 | WAS | +170 | 60.0% | 64.5% | ▼ -4.5 pp | 0.012 | lost | -1.00 | -0.30 | -4.3% |
| 2026_02_NYG_LA | 2 | NYG | +240 | 65.0% | 71.8% | ▼ -6.8 pp | 0.020 | lost | -1.00 | -1.30 | -16.3% |
| 2026_03_ATL_GB | 3 | ATL | +235 | 67.0% | 71.4% | ▼ -4.4 pp | 0.011 | won | +2.35 | +1.05 | +11.7% |
| 9 plays | 3 won | mean |edge| 7.0 pp | 0.20 | +1.05 | +1.05 | +11.7% |
Flat 1u: +1.05u on 9u risked → ROI +11.7%. 0.25 Kelly: -0.059u on 0.20u risked → ROI -28.8%.
9 plays cannot establish an edge. Detecting a 3% ROI edge needs about 8,722 bets, so treat this ROI as a log of what happened, not as evidence of skill in either direction.
The line moved our way on 7/41 (17%), and was unchanged on 33.
Correlation between the blind estimate and the market over 41 games: r = 0.638. Near 1 means it is reproducing the market; near 0 means it is either finding something independent or ignoring the evidence — the Brier table above tells you which.
| Record | Flat P&L | Flat ROI | 0.25 Kelly ROI | Mean edge | Current streak | Longest W / L |
|---|---|---|---|---|---|---|
| 3–6 | +1.05u | 11.7% | -28.8% | 7.0% | 1 win | 1 / 4 |
| Scenario | Flat P&L over these plays | Meaning |
|---|---|---|
| If the model's probabilities are right | +1.31u | positive by construction — it is why it bet |
| If the market's prices are right | -0.37u | negative — this is the vig |
| What actually happened | +1.05u |
| Metric | Actual | Expected | ± 1 s.d. | z | Verdict |
|---|---|---|---|---|---|
| Wins | 3 | 3.9 | 1.5 | -0.63 | entirely normal |
| Flat P&L | +1.05u | +1.31u | 4.13 | -0.06 | entirely normal |
A losing streak of 4 had a 33% chance of happening anyway, given these probabilities. Streaks are the statistic humans over-read the most — the z-scores above are the ones to trust.
Expected wins come from a Poisson-binomial: our plays are independent trials with different probabilities, so the expectation is the sum of them and the variance the sum of p(1−p). Several z-scores are shown, so treat any single one crossing 2 loosely.
Counted per game, not per sample. Unlike everything above, this is directly actionable.
| Games that wanted it | Missing data |
|---|---|
| 11 | Market line withheld |
| 6 | Market spread/line withheld |
| 4 | No injury reports filed |
| 4 | Special teams EPA and hidden-yardage metrics |
| 4 | No injury report filed for either team this week |
| 3 | No pre-game market line provided |
Flat betting is up +0.50 units on 9 plays (ROI 5.6%), and that means nothing yet. wins_z = -0.667 and pnl_z = -0.193, both "entirely normal" — a 3-6 record against expected_wins = 3.975 is less than one standard deviation (wins_sd = 1.46) off. The two hypotheses we are trying to separate are expected_pnl_model = +1.28 and expected_pnl_market = -0.37; they are 1.65 units apart against pnl_sd = 4.05. Our +0.50 sits almost exactly midway, roughly 0.4 SD from either line. The honest answer is that 9 plays cannot distinguish "we have edge" from "we are paying vig." brier_model = 0.2247 vs brier_market = 0.2218 — a gap of 0.0029 over 32 games, well inside noise, but it is the correct direction to worry about: we are not yet beating the closing line on accuracy.
Nothing this week on the model, the prompt, or --edge-threshold 0.04. No result here clears |z| = 2.
One reporting bug worth fixing, though: the misses list is ranked by Brier loss and is silently mixing in winning bets. All six misses have negative edge — the model was below market on every one. 2026_02_CLE_TB (edge -0.047) and 2026_01_SF_LA (edge -0.043) cleared the threshold as away-side plays, and the home team lost, so those were wins being displayed as failures. The other four (-0.020, -0.007, -0.034, -0.007) were never plays at all. The miss table should flag played and side or it will keep inviting exactly the tuning error this report exists to prevent.
kelly_roi = -32.3%. On kelly_staked = 0.203 units that is -0.066 units. Rounding error dressed as a catastrophe.p_loss_streak_this_long = 0.296 — it happens in roughly three seasons out of ten by chance alone.sample_spread was 0.01–0.08 on every one: the model was internally consistent and the market agreed with it (it priced them higher than we did in all six). Category (c), variance, across the board. No case here shows a bad inference from data it actually had.pbp already in the warehouse over the same trailing-8 window.Ignore the top of common_data_gaps: "Market line withheld" and its variants (11 + 6 + 3 + 3 + 3) total ~26 and are the --blind pass working as designed. Supplying it would defeat the experiment.
Generated 2026-09-23 09:37 UTC by Claude Code from performance.py --json. Analysis of model output, not betting advice.