An honest and objective assessment of poll performance

[JohnStOnge]

Ok I was stuck with not much to do today so I did an verification exercise with the pre election polling just like I would do with any other statistical assessment. Bottom line is that if you were going on the polling results available and sticking to the rule you would have said Clinton will win the popular vote but there was not sufficient evidence to say either candidate would win the electoral college. However, I must say that there was indeed a bias towards Clinton. Not a big one. But it is there. Among the States I was able to do the polls, on average, over estimated Clinton's margin by 2.7 percentage points and it was "significantly" greater than 0. That's not a big bias, but even a small one like that can be big when you've got a guy winning a number of key States by less than 2 points.

One interesting thing is that I decided before I started to only look at polling that started after October 28. That's because that's when Comey dropped his bomb about Weiner's computer. I figure that's an obvious significant event that could've had an impact so polling including responses prior to that shouldn't be used. I also made a rule that I had to have at least two polls to look at.

Well... that created an immediate problem because only 17 States were represented by polling meeting those criteria. So if I stuck to my rules I'd had to have said that 33 States plus DC accounting for 304 electoral votes were unknown. I went ahead and tried to look at the latest polling in those States and make judgements. For instance if you look at Maryland and the last poll was conducted September 27 - 30 and it had Clinton up by 36 percentage points I went ahead and put Maryland in the Clinton column.

I ended up saying there was sufficient evidence to say Clinton would get at least 231 electoral votes and Trump would get at least 104 with 201 electoral votes uncertain. So too close to call on the electoral college front.

There was a problem with the polling in that 6 results fell outside of the 17 margins of error I did and no more than 2 should have due to random chance. Also, all 6 of those results involved Trump "over performing."

There were two states, Michigan and Wisconsin, such that I would have said the polls provided sufficient evidence to say that Clinton would win and she did not.

Nationally, I would have said the polls showed Clinton would win the popular vote and she did. However, I'd have said she'd win by from 2.3 to 4.4 percentage points and she actually won it by 2.1 percentage points. So, again, a small bias.

Bottom line, though, is that if you were honestly assessing the polls you would have said Clinton will win the popular vote but the Electoral College is too close to call.

[BDKJMU]

I don't think anyone read your 8 paragraphs of drivel..

[93henfan]

I don't think anyone read your 8 paragraphs of drivel..

I did. Cliff's Notes version:

Trump
Triggered
Triggered
Triggered
Triggered
Triggered
Triggered
Triggered

[Pwns]

Ok I was stuck with not much to do today so I did an verification exercise with the pre election polling just like I would do with any other statistical assessment. Bottom line is that if you were going on the polling results available and sticking to the rule you would have said Clinton will win the popular vote but there was not sufficient evidence to say either candidate would win the electoral college. However, I must say that there was indeed a bias towards Clinton. Not a big one. But it is there. Among the States I was able to do the polls, on average, over estimated Clinton's margin by 2.7 percentage points and it was "significantly" greater than 0. That's not a big bias, but even a small one like that can be big when you've got a guy winning a number of key States by less than 2 points.

One interesting thing is that I decided before I started to only look at polling that started after October 28. That's because that's when Comey dropped his bomb about Weiner's computer. I figure that's an obvious significant event that could've had an impact so polling including responses prior to that shouldn't be used. I also made a rule that I had to have at least two polls to look at.

Well... that created an immediate problem because only 17 States were represented by polling meeting those criteria. So if I stuck to my rules I'd had to have said that 33 States plus DC accounting for 304 electoral votes were unknown. I went ahead and tried to look at the latest polling in those States and make judgements. For instance if you look at Maryland and the last poll was conducted September 27 - 30 and it had Clinton up by 36 percentage points I went ahead and put Maryland in the Clinton column.

I ended up saying there was sufficient evidence to say Clinton would get at least 231 electoral votes and Trump would get at least 104 with 201 electoral votes uncertain. So too close to call on the electoral college front.

There was a problem with the polling in that 6 results fell outside of the 17 margins of error I did and no more than 2 should have due to random chance. Also, all 6 of those results involved Trump "over performing."

There were two states, Michigan and Wisconsin, such that I would have said the polls provided sufficient evidence to say that Clinton would win and she did not.

Nationally, I would have said the polls showed Clinton would win the popular vote and she did. However, I'd have said she'd win by from 2.3 to 4.4 percentage points and she actually won it by 2.1 percentage points. So, again, a small bias.

Bottom line, though, is that if you were honestly assessing the polls you would have said Clinton will win the popular vote but the Electoral College is too close to call.

I get what you have been saying about the polls.

The problem does not seem to come from predicting individual states, but integrating all the data from each state to get a probability of a Clinton win. People like Nate Silver were saying Clinton had a 90%+ chance on the EC, and given Trump got 300+ EC votes it's a tough sell to say that that was correct.

If you took whatever model or estimation method Nate Silver used to estimate a 90% chance of a Clinton win I would bet the calculated probability that Trump would more than 300 EC votes I bet it was pretty small, like less than one-tenth of one percent. Which should call into question his entire method of getting the outcome probability.

Back around 2012 or so I tried to tell BlueHen86 and other people here that Nate Silver is just an amateur doing voodoo math and his predictions of the EC vote in 2012 really weren't that remarkable. Now I'm vindicated for that.

[CAA Flagship]

It was the economy.

[JohnStOnge]

People like Nate Silver were saying Clinton had a 90%+ chance on the EC, and given Trump got 300+ EC votes it's a tough sell to say that that was correct.

Actually Nate Silver's site, 538, put the probability of a Clinton win at 71.8% in their final "polls plus" forecast and at 71.4% in their final "polls only" forecast. They're till up at https://projects.fivethirtyeight.com/20 ... -forecast/.

That's basically saying you're at around the 71 or 72% confidence level. By convention one would NEVER make a prediction based on that low a level of confidence. It was too close to call. And I pretty much said that in a post I made at http://www.championshipsubdivision.com/ ... start=2575. Second post from the bottom of the page. I wrote:

It's not over by any means. To put it into terms we're familiar with a 70.3% chance of winning is about equivalent to football team being an 8.5 point favorite. So by no means a done deal. I feel better than I would if the probabilities were reversed. But not at ease at all.

And actually I was off on the football analogy. 70.3% (which was what the forecast was at the time) is about equivalent to being a 7 point favorite. That's based on a logistic regression I did back in August 2015 using 2003 - 2014 data from the web page at https://www.teamrankings.com/ncf/odds-history/results/.

To put THAT into perspective, Alabama opened as a 7 to 8 point favorite over Clemson, depending on the book, for the 2017 College Football Championship game (http://www.cbssports.com/college-footba ... r-clemson/).

Yes Clinton was the favorite. But it wasn't like Alabama playing Vanderbilt. There was a LOT of doubt and it wasn't the huge upset it's generally portrayed as. And, as you can see, I was saying the night before the election that it was "...by no means a done deal" and that I was "...not at ease at all."

I'm not just saying this stuff after the fact.

[kalm]

It was the economy.

And Clinton being a horrendously untrustworthy candidate.

Good thing Trump has appointed a few billionaires and a Goldman Sachs guy to right the ship for the little guy...

[JohnStOnge]

Ok here's a simple one for you guys that don't like to read. I just took the raw averages at the top of the page on relevant polls for all the States RealClearPolitics has polls for. Just wanted to see what one would have predicted on the night before the election from the polls straight up without worrying about "statistical significance" and without having any other information to go on.

And what you'd have predicted is that Hillary Clinton would win the popular vote but the Electoral College winner couldn't be predicted. That's because Hillary Clinton led in the average of overall vote polls by 3.3 percentage points and there are no polls for five jurisdictions with 22 electoral votes. The totals for States for which polls were available are Clinton 266, Trump 250. So, again, the conclusion one would reach just looking at the polling and not introducing other sources of belief and information is "can't make a call."

Overall, just straight up, the candidate with the polling average lead won in 42 of 46 cases (91%) for States where polling data are available. There was a bias towards Clinton in the State by State polls, no doubt. On average her margin was over-predicted by 3.5 percentage points. And she lost three States where she had the higher polling average while Trump lost only one.

But if you're one of those people who think what in the grand scheme of things was a relatively small bias in the polling for that election means you can discount polls now that suggest things you don't want to believe you are kidding yourself; especially when you're looking at national polls because those weren't far off at all.

But that's what Trump is attempting to foster. A central aspect of his approach is disseminating misinformation. And he wants to discredit anything that contradicts the avalanche of false information he discharges.

[Pwns]

People like Nate Silver were saying Clinton had a 90%+ chance on the EC, and given Trump got 300+ EC votes it's a tough sell to say that that was correct.

Actually Nate Silver's site, 538, put the probability of a Clinton win at 71.8% in their final "polls plus" forecast and at 71.4% in their final "polls only" forecast. They're till up at https://projects.fivethirtyeight.com/20 ... -forecast/.

That's basically saying you're at around the 71 or 72% confidence level. By convention one would NEVER make a prediction based on that low a level of confidence. It was too close to call. And I pretty much said that in a post I made at http://www.championshipsubdivision.com/ ... start=2575. Second post from the bottom of the page. I wrote:

It's not over by any means. To put it into terms we're familiar with a 70.3% chance of winning is about equivalent to football team being an 8.5 point favorite. So by no means a done deal. I feel better than I would if the probabilities were reversed. But not at ease at all.

And actually I was off on the football analogy. 70.3% (which was what the forecast was at the time) is about equivalent to being a 7 point favorite. That's based on a logistic regression I did back in August 2015 using 2003 - 2014 data from the web page at https://www.teamrankings.com/ncf/odds-history/results/.

To put THAT into perspective, Alabama opened as a 7 to 8 point favorite over Clemson, depending on the book, for the 2017 College Football Championship game (http://www.cbssports.com/college-footba ... r-clemson/).

Yes Clinton was the favorite. But it wasn't like Alabama playing Vanderbilt. There was a LOT of doubt and it wasn't the huge upset it's generally portrayed as. And, as you can see, I was saying the night before the election that it was "...by no means a done deal" and that I was "...not at ease at all."

I'm not just saying this stuff after the fact.

Okay fine, but the same problem still holds. You aren't talking about a team that's an 8-point favorite that won by 3. The dog won by at least 2 touchdowns.

Just using the probabilities on Silver's map, the chances of Trump winning WIsconsin, Michigan, Pennsylvania, Ohio, North Carolina, and Florida is 0.1%.

If you're doing model-building in your practice and the chance of something happening is that small, do you assume that the planets aligned or that it's just a s**tty model?

My guess is that correlation across states isn't well-accounted for, which is why the number of states where the outcome was outside the margin of error is so high.

[JohnStOnge]

Okay fine, but the same problem still holds. You aren't talking about a team that's an 8-point favorite that won by 3. The dog won by at least 2 touchdowns.

Just using the probabilities on Silver's map, the chances of Trump winning WIsconsin, Michigan, Pennsylvania, Ohio, North Carolina, and Florida is 0.1%.

If you're doing model-building in your practice and the chance of something happening is that small, do you assume that the planets aligned or that it's just a s**tty model?

My guess is that correlation across states isn't well-accounted for, which is why the number of states where the outcome was outside the margin of error is so high.

I disagree with the idea that it was like the dog winning by two touchdowns. The wins in those swing states were all very narrow. It's the way it works with our Electoral College system. If you win by 0.7 percentage points in Pennsylvania as Trump did you get 100% of the electoral votes. It was an extremely close election in real terms.

The "model" in this case before the fact did not indicate sufficient evidence to make a call. It's always possible after the fact to find things that happened that were unlikely.

[JohnStOnge]

Just using the probabilities on Silver's map, the chances of Trump winning WIsconsin, Michigan, Pennsylvania, Ohio, North Carolina, and Florida is 0.1%.

If you're doing model-building in your practice and the chance of something happening is that small, do you assume that the planets aligned or that it's just a s**tty model?

My guess is that correlation across states isn't well-accounted for, which is why the number of states where the outcome was outside the margin of error is so high.

First let me say that it's nice to actually have a substantive discussion on this topic. I thought about your question and I think the situation you describe is a result of the bias in the polling estimates. I don't think the bias was that big. When I tried to design my own approach I estimated it at somewhere in the range of 1.5 through 5.6 percentage points. In an article on post election analysis of poll performance a pollster quoted in a 538 article put it at "...a 4-point or so national miss..." (https://fivethirtyeight.com/features/th ... sters-why/).

I think that might make a big difference when assigning confidence levels State by State. For example: Left on my own prior to the election I would have assigned 50% confidence to Trump winning Florida. That's pretty close to 538 assigning 44.9% confidence in Trump winning the State in its "polls only" forecast. But that's assuming 0 bias.

If I would have assumed bias at just 1.8 percentage points (that's a 95% one sided "no less than" estimate for Florida) rather than 0 I would've assigned 83% confidence to Trump winning the State.

Nevertheless, even if one wrongly assumed that the polls were theoretically perfect with 0 bias there was not sufficient evidence in the polling data to predict a winner in the Electoral College. And, if anything, people should have had enough sense to know that there's no way pollsters are going to get theoretically perfect polling estimates with 0 bias. They can't get true probability samples. They have to deal with non response. They have to make assumptions about who is actually going to vote. There's just no way the polling data supported confidently declaring a winner and it's not the pollsters' fault if people leaped to unsubstantiated conclusions.

Also, it would be a mistake to dismiss something like the body of polling showing that Trump has a historically low job approval rating for someone at this point of their Presidency based on the idea that "the polls got it wrong" in the election; especially when those are national polls and the average of national polling reported by RealClearPolitics just prior to the election had Clinton's popular vote margin only 1.1 percentage points higher than the actual result (3.2 vs. 2.1 percentage points).

And you can see what Trump and his team are trying to do. They're putting out a bunch of bullshit and trying to convince people to disbelieve information that shows that bullshit to be what it is. Whether it's trying to get people to ignore polling estimates (except when they get an outlier like Rasumussen that appears to support the bullshit) or attacking legitimate news reports as "fake news" that's what they're doing.

We can only hope that, eventually, people outside of the hopeless crowd including people like the Cardboard Trump guy will get wise to what's going on.

[ASUG8]

Clinton lost the electoral. It's over.

[Pwns]

You could be right about the bias....and the bias is going to be correlated across states...if you undersample or oversample a certain type of voter in one state there's no reason not to expect you won't do it in other states. Not only that, but the bias is not something you can just make an estimate based on historical data and then apply that to future elections. Sampling methods can change, voting methods can get better or reach more people, and voter turnout of various groups is going to ebb and flow depending on who the candidates are. This makes EC vote prediction hard.

Like I said, I'm just not convinced the people trying to predict the EC vote had any clear picture of what it might look like. In 2012 there were just fewer states that were up in the air.

[GannonFan]

I stopped reading when I saw JSO was the author of a thread that started with the words "An honest and objective...". If I was drinking something I'm sure I would've spit it out at the moment of reading that phrase.

[Ibanez]

[Rob Iola]

My wife says I don't need Viagra.

Oh wait, this is about Trump you say? Nevermind...

[Ibanez]

My wife says I don't need Viagra.

Oh wait, this is about Trump you say? Nevermind...

[JohnStOnge]

I stopped reading when I saw JSO was the author of a thread that started with the words "An honest and objective...". If I was drinking something I'm sure I would've spit it out at the moment of reading that phrase.

I am always honest and objective. And I think you really know that.

That's why I was saying on the night before the election that I was very nervous about the outcome. I wasn't seeing what I would like to have seen in the polling data, which I was objectively assessing, and I was honest with both myself and others about that.

Also why, after repeatedly defending Republican voters on this board over many years and after having voted Republican in every Presidential election of my lifetime, I determined that THIS time the Republicans candidate was unacceptable, that the Republican Party does have a problem with its voter base, and that I need to re-assess my historical defense of that voter base.

[YoUDeeMan]

I stopped reading when I saw JSO was the author of a thread that started with the words "An honest and objective...". If I was drinking something I'm sure I would've spit it out at the moment of reading that phrase.

I am always honest and objective. And I think you really know that.

No, JSO...you lost any claim to that a while ago.

[JohnStOnge]

I am always honest and objective. And I think you really know that.

No, JSO...you lost any claim to that a while ago.

Just a blind insult. You have no actual basis for making that statement.

I'll give you an example of something we've seen repeatedly that provides an actual basis for saying someone's not being objective: When you see someone rationalize around something like the Politifact file on Trump to deny that he's a liar of historic proportions by saying it's a "liberal" effort. That's an example of failing to be objective.

You're not going to find something like that with me.

[JohnStOnge]

BTW there's another way to demonstrate that the election was too close to call. That's just by looking at how close the final margins in certain States were:

Florida 1.2% (Trump)
Michigan 0.3% (Trump)
New Hampshire 0.4% (Clinton)
Pennsylvania 0.7% (Trump)
Wisconsin 0.8% (Trump)

There is just no way polls are going to have the "resolution" to see margins that small. Take a look at the 538 pollster ratings at https://projects.fivethirtyeight.com/pollster-ratings/. Click on the "538 Grade" column heading to sort by grade. Look at the most highly rated polls (A+). Look at the "Simple Average Error" column. Note that the values for the BEST polls range from 3.0 through 5.5 percentage points. That was based on historical results collected 1998 through August 2016.

There's just no way polls can possibly have the "precision" to predict the result for a State where the actual margin is less than 2 percentage points. Expecting otherwise is not being realistic.

[kalm]

BTW there's another way to demonstrate that the election was too close to call. That's just by looking at how close the final margins in certain States were:

Florida 1.2% (Trump)
Michigan 0.3% (Trump)
New Hampshire 0.4% (Clinton)
Pennsylvania 0.7% (Trump)
Wisconsin 0.8% (Trump)

There is just no way polls are going to have the "resolution" to see margins that small. Take a look at the 538 pollster ratings at https://projects.fivethirtyeight.com/pollster-ratings/. Click on the "538 Grade" column heading to sort by grade. Look at the most highly rated polls (A+). Look at the "Simple Average Error" column. Note that the values for the BEST polls range from 3.0 through 5.5 percentage points. That was based on historical results collected 1998 through August 2016.

There's just no way polls can possibly have the "precision" to predict the result for a State where the actual margin is less than 2 percentage points. Expecting otherwise is not being realistic.

So with that wide of range, the problem seems to be publishing the percentage chance of winning. The media believed that a 4% lead in certain states meant a 75% chance of winning.

Still the pollster's and media's fault.

[Gil Dobie]

Bottom line, though, is that if you were honestly assessing the polls you would have said Clinton will win the popular vote but the Electoral College is too close to call.

Did you tell us this prior to the election or is it hindsight?

I didn't vote for either one or Jill Stein, and I'm glad I didn't.

[JohnStOnge]

Bottom line, though, is that if you were honestly assessing the polls you would have said Clinton will win the popular vote but the Electoral College is too close to call.

Did you tell us this prior to the election or is it hindsight?

I didn't vote for either one or Jill Stein, and I'm glad I didn't.

I didn't comment on the popular vote prior to the election but I did post on the night prior to the election that the electoral college situation was uncertain. "Not a done deal at all" is the specific terminology I used.

I wasn't even really looking at trying to call the popular vote winner. I was looking at the confidence levels associated with trying to say one candidate or the other would win the Electoral vote and it was clearly a situation in which there wasn't sufficient evidence to make a call one way or the other.

[JohnStOnge]

So with that wide of range, the problem seems to be publishing the percentage chance of winning. The media believed that a 4% lead in certain states meant a 75% chance of winning.

Still the pollster's and media's fault.

I don't think it was the pollsters' fault. One thing is that when people say "75% chance" of winning that's not really accurate. It's "75% confidence" of winning. I won't even go into why the distinction is important but it is.

The main thing is that 75% confidence is nowhere near enough to make a call. By convention the standard confidence level necessary to make a call is 95%. You could stretch it and maybe go down to 90%. But there's no way anybody who is really following the rules is going to go below 90%.

And yes I realize that most in the media don't understand that. In fact I think even people who are the faces of polling organizations like Frank Luntz and Kellanne Conway probably don't understand that. I say that because of some of the stuff I've seen them say. I think they are like manager people who manage polling operations and really don't have a clue with respect to the statistics aspect. Could be wrong but that's the impression I get watching/listening to them talk.