Sagarin vs. Mvemjsunpx during the playoffs

[JohnStOnge]

You guys probably won't read this but since I like doing probability stuff I'll post it anyway. It's another illustration of the effect of making a whole bunch of comparisons then deciding what is "best" after the fact.

Let's say we were comparing Sagarin to 107 other people since as I read it that was the number of people in the AGS thing. And let's say all of the people are exactly as good as each other in picking games. Now let's say you know Sagarin is a little bit better in that, going in, it's got a 60:40 chance of beating any particular person.

Then you say you're going to have Sagarin go up against each person. Going in, you could be 95% confident that at least 36 people will beat Sagarin. And the probability that Sagarin would be every one of the 107 persons, even though in that scenario we know it is indeed better than each individual person, is so small that the Excel I'm using can't calculate it. Excel just outputs 0.

That's an illustration of how pointless trying to determine which system is "best" by throwing a bunch into the pot then looking at the results after the fact is.

[89Hen]

[Mvemjsunpx]

Most people would be using these for spread betting purposes, though, and the Sagarins didn't do too well there (as I mentioned earlier). They weren't nearly as bad as the Masseys, but still a bit underwater. I don't know if anyone has tracked these rankings in previous years, however, so I don't know how they did then. I did track ebemiss's NationalSportsRankings simulator last season and they ended up slightly in the black, but I don't think he puts current season teams in until about the beginning of October.

My theory on this remains the same: college football seasons have too few games to make computer rankings accurate enough to be real valuable. Yeah, these rankings do better than average or even solid human pickers, but not well enough to beat real knowledgeable folks.

[JohnStOnge]

I think it's looking at two different things. You can go to http://www.thepredictiontracker.com/ncaaresults.php" onclick="window.open(this.href);return false; and see how various power rating systems did against the spread. But they are not designed to try to beat the spread. They are designed to try to predict the spread.

What you look at to see how well they did in that regard is the Absolute Error, Bias, and Mean Square Error. That's all at that page too. Bias is the average of the predicted margin - the actual margin for each game. Absolute error is the average of absolute value of predicted margin - absolute value of actual margin and mean square error is the average of (predicted error - actual error) squared. The ideal for all three measures is 0.

It'd be interesting to see how people who are good at beating the spread more often than not do by those measures. They wouldn't just be saying they're going to pick a team plus or minus the points. They'd be giving their own specific prediction of what the margin would be. Another thing is that they would have to do it over all the games without cherry picking. I don't know if it's always like this but when I hear people on the radio and such picking games they always pick a few they like. It's be interesting to see if they could beat the updated line or something like Sagarin in terms of getting closer to 0 in terms of the three referenced measures or in terms of percent correct game winners picked. Then the next step is that you'd have to do statistical tests to see if any differences are "significant."

[JohnStOnge]

I do know that it's not uncommon for human polls to rank teams ahead of other teams even though everybody thinks the lower ranked team is better. Or at least most people do. And example of that happened just a little while ago with the FBS National Championship Game. Clemson was ranked #1 but it was pretty clear that just about everybody thought going in that Alabama was better. The line also reflected that in having Alabama as the favorite.

That's one reason I like good power rating systems. They don't have that kind of thing in them. To me there's often kind of some kind of cultural "ethical" thing in polls where people will rank team B over team A even though if you ask them who they think would win if the two played they'll say it's team A. Human polls also, I think, put too much emphasis on records without taking strength of schedule into account. At least during the regular season. In FCS I think they put too much emphasis on how far teams go in the playoffs. Like for instance, once again, I don't think there's any way Sam Houston State was one of the top 4 teams in FCS this past season but they're ranked 4th in both the STATS and FCS Coaches final polls because they made it to the semifinals. When you look at their entire body of work this season there's no way you would come up with that. If you look at the composite of rating systems at http://www.championshipsubdivision.com/ ... &p=1030160" onclick="window.open(this.href);return false; they're rated 14th and I think that's more realistic. And yes I know you can find some systems that are used in the composite that look bad but I still think using it is better than using any single rating system.

[Mvemjsunpx]

I think it's looking at two different things. You can go to http://www.thepredictiontracker.com/ncaaresults.php" onclick="window.open(this.href);return false; and see how various power rating systems did against the spread. But they are not designed to try to beat the spread. They are designed to try to predict the spread.

I fail to see how those two things are different. If one prediction's spread is on one side of the "official" spread, then it's predicting the spread is it not? And gambling is one of the primary reasons people look at these things. I guess the assessment of how accurate the ranking is might not be the same between the either/or betting line and the pinpoint/margin-of-error analysis, but—with enough trials—you'd expect it have the same relative accuracy.

[JohnStOnge]

I fail to see how those two things are different. If one prediction's spread is on one side of the "official" spread, then it's predicting the spread is it not?

It depends on what they do. Like right now if I'm reading right the line at http://www.sportsbettinglines.com/patri ... nder-odds/" onclick="window.open(this.href);return false; for today's AFC Championship game is Patriots by 14 (which surprises me...that's a lot...that's why I'm wondering if I'm reading right). If someone says "I'm taking Denver plus the points" there's no way to compare them to the criteria I'm talking about. If the Patriots win by 8 we can say they beat the spread but they didn't give us a prediction as to what the actual margin is going to be or as to who will win the game.

If instead of just saying they're taking Denver and the points they said "I think Denver is going to win straight up by 3 then they've given us a specific prediction of the type I'm talking about. Then let's say the actual outcome is Patriots winning by 13. Well, now the person who picked Denver in the game beats the spread because Denver is getting 14 points. They get paid. But the line was much closer in terms of predicting the outcome. It missed by one point while the person who picked Denver by 3 missed by 16. Also the person who picked Denver by 3 was wrong on the winner while the line was correct.

So you can see that beating the spread and being more accurate in predicting the outcome of the game can be two different things.

.

[Mvemjsunpx]

I fail to see how those two things are different. If one prediction's spread is on one side of the "official" spread, then it's predicting the spread is it not?

It depends on what they do. Like right now if I'm reading right the line at http://www.sportsbettinglines.com/patri ... nder-odds/" onclick="window.open(this.href);return false; for today's AFC Championship game is Patriots by 14 (which surprises me...that's a lot...that's why I'm wondering if I'm reading right). If someone says "I'm taking Denver plus the points" there's no way to compare them to the criteria I'm talking about. If the Patriots win by 8 we can say they beat the spread but they didn't give us a prediction as to what the actual margin is going to be or as to who will win the game.

If instead of just saying they're taking Denver and the points they said "I think Denver is going to win straight up by 3 then they've given us a specific prediction of the type I'm talking about. Then let's say the actual outcome is Patriots winning by 13. Well, now the person who picked Denver in the game beats the spread because Denver is getting 14 points. They get paid. But the line was much closer in terms of predicting the outcome. It missed by one point while the person who picked Denver by 3 missed by 16. Also the person who picked Denver by 3 was wrong on the winner while the line was correct.

So you can see that beating the spread and being more accurate in predicting the outcome of the game can be two different things.

.

The spread isn't quite that simple, though. 5dimes (which is what I used for my analysis) offers multiple spread lines for all FBS games (& most FCS postseason games) & straight-up money line bets for all games. If the default spread was 10 but the Sagarins picked the favorite to win by 13.2, then I had them take the favorite to cover minus-13 with a beneficially lopsided money line (assuming a minus-13 was there; otherwise they'd just take the highest available). If the Sagarins picked the favorite to lose, I would have them take the straight-up money line bet rather than the spread. This whole idea makes the spread more of a mix between the two concepts, I guess. I do think you have some good points, though—my analysis wasn't so much about which system was more accurate in a vacuum but about which system was most beneficial for gambling purposes.