Poll of Polls: how did the prognosticators perform

Remember all those CIS XC predictions? How’d they match up against what really went down? Who was the best at predicting? The TNFNorth Panel? Evan Andrin? The CIS coaches? Montreal Endurance?

Here, in Chart form, are the four pre-championship polls, stacked up against the actual results.

So if we score it in the following arbitrary way, 1 point per correct prediction, we can see that on the women’s side

Coaches: 2
TNFNorth: 2
P and P: 0

Not very good predicting. Everyone got #1 and #2 (hard to miss), but McGill’s hard fall and Western’s rise messed up the results for everyone. To be fair, I made my predictions after knowing for sure Jewett was out, but everyone missed my alma mater Warriors (Athenas). Good job coach Sheilds. And only a couple people at TNFNorth thought Dalhousie would be in the top ten, but not enough for them to make it into the final poll.

On the men’s side:

Coaches: 3
TNFNorth: 5
P and P: 2

I’m not sure that’s much better. I don’t know why Lakehead was the popular (and correct) choice for 10th, considering all that went on above that. It seems like St. FX did not perform as the pollsters expected. Everyone had 9 out of the top 10 teams correct, if not in the right order. X fell short, and home course Mustangs ran their best race of the year. Just goes to show how much rankings are worth.

If we break down the TNF North Poll, to give the voters the credit they deserve (or not), here’s how they would have scored:

Jeff Barr: 4
Rich Lehman: 2
Chris Moulton: 3
Jeremy Walsh: 2
Steve Gersten: 4
John Lofranco: 4 (I voted slightly differently than in my MTLE posting)

And here are the men:

Jeff Barr: 0
Rich Lehman: 1
Chris Moulton: 1
Jeremy Walsh: 2
Steve Gersten: 2
John Lofranco: 2

Oddly, the individual scores for the women are much better than the aggregate, but on the men’s side, it’s the opposite: horrible predicting by each person ended up calling 50% of the race, the best of any predictor. I suppose that is something Nate Silver could explain.

So what do we learn from this? Don’t trust the polls? Well not quite. While I lack the statistical chops to explain it, I think most polls did ok, when you consider that one or two teams over (Laval, Western) or under (McGill, Regina) performed relative to what was expected. The order was generally not bad, once you factor that in. But, has most had been saying all year, it was pretty tight from 2nd on down on the men’s side, and certainly from 3rd to 7th on the women’s side was really a difference of one runner here or there.

Anyway, this post is only to add some accountability to a season of polling. Maybe next year we will come up with a 538esque way of analyzing all of this. Any ideas/stats geeks are welcome to contribute!