Until this week, a start call counted as a hit if the player beat the median weekly score at his position. The critique: that bar gives full credit for calling "start" on a player who merely has an adequate week — and adequate weeks are common for the players analysts talk about most.
We tested it. We rebuilt the bar so that a start call only counts as a hit if the player actually has a startable week — a top-12 finish at QB or TE, top-24 at RB or WR, computed fresh each week from the full player population. Seventy-two thresholds, one per position per week, now frozen for the season.
Then we regraded all 1,875 resolved 2025 start/sit claims in the corpus against the new bar.
The critique held up: 73.8% of start-call hits under the old bar were on players who also cleared the new one. The other 26.2% were hits on players who never had a top-12/24 week — adequate-not-elite performances that had been getting full credit. Roughly a quarter of the old system's start "hits" shouldn't have counted.
Under the old bar, our internal numbers showed start calls as the stronger skill — 61.4% on starts against 47.3% on sits, a finding we were preparing to publish. It was an artifact of the bar. Under a startable-week standard, the picture inverts completely:
We didn't trust that reversal on its face, because a harder bar mechanically helps sit calls: more players finish below a top-24 line than below a median. So before publishing, we tested the sit number against two separate baselines a skill-less caller could achieve.
Test 1 — coin flip on the same players. We ran a Monte Carlo assigning random start/sit directions to the exact (player, week) pairs analysts actually called sits on. A random caller hits 50.5%. Real sit calls hit 62.8% — a 12.3-point edge that can't be explained by which players got discussed.
Test 2 — blind pessimism. Across all 1,875 resolved claims, 55.1% of discussed players finished below the startable line. So a strategy of calling "sit" on every single player analysts mention — no judgment at all — scores 55.1%. Real sit calls beat that by 7.7 points. Analysts aren't just benefiting from a bar that favors sits; they know which discussed players to sit.
Two different tests, two different assumptions, same answer. The sit skill is real.
One honest caveat: 296 sit calls is a much smaller sample than 1,579 starts. The edge is large enough to clear both baselines comfortably, but the sample will roughly quintuple by season's end, and the number will move. We'll report where it lands.
The same two tests on start calls: 46.4% observed, against a 50.0% random-direction baseline (−3.7) and a 44.9% always-start baseline (+1.5). One test slightly below chance, one slightly above — straddling zero. In fairness to the analysts, the players they select for start calls are a harder pool than the discussed-population average, so the raw number understates their selection judgment somewhat. But whatever start-calling skill exists in this corpus, it is too small for either test to detect.
The pattern is intuitive once you see it: telling you to start the players everyone already starts carries no information. Telling you to bench someone you were planning to start — that's a real claim, made against the grain, and it's where the measurable skill in this corpus lives.
The full regraded leaderboard is live at scoutrank.app. Every score now reflects the startable-week bar, every receipt shows the actual per-week line the call was graded against, and each publisher's start-call and sit-call accuracy is broken out separately.
Two commitments alongside it. First, the bar is frozen: no changes to scoring thresholds until the season ends. Corrections to data, yes — changes to the standard, no. Second, everything above is versioned as SR-2026.4 on our methodology page, including the full changelog — this is our second public methodology correction, and like the first, it exists because someone outside the project pushed on the right thing.
If you see something that looks wrong, that's the mechanism working. Push on it.
Every number in this article traces back to a receipt. See the live board, or how the bar is actually computed.