Over Interpreting Polls.

[JohnStOnge]

Every time elections come around I have to shake my head over the way pundits look at modest differences in day to day polls then launch into analyses of why things changed. Fact is, a lot of the time, the polls don't provide sufficient evidence to say anything changed.

For purposes of illustration, I ran a simulation in which the true percent support in the population was 40% for Candidate A and 40% for Candidate B with 20 percent undecided. I did 10 runs of random samples of 1000 each. This is like taking 10 political polls consecutively to represent the same situation.

Among the 10 runs the "best" poll result I got for Candidate A was A getting 41.0% to B getting 36.6% with 22.4% undecided. The "best" poll result for Candidate B was B getting B getting 42.6% to A getting 36.9% with 20.5% undecided. So you're talking about a 10.1percentage point swing between the two with A being ahead by 4.4% in one poll and B being ahead by 5.7% in the other.

But the reality is that there was NO DIFFERENCE between the actual percentages in the population. The difference in the simulated poll results is just random sampling error.

It is SO ridiculous that people see changes in these polls and start acting as though that means there necessarily was an actual change in the true percent support for each candidate in the population. But I guess it gives us something to watch on TV and talk about.

[TwinTownBisonFan]

24 hour cable news covers politics like ESPN covers the NFL... reporting on the most trivial of things to death because it means ratings.

Head-to-head polling is largely useless - especially without crosstabs. The polling we do internally is quite a bit different. We tend to focus on message testing and issue testing. Helps us find out which voters we target and how.

[citdog]

Helps us find out which voters we target and how.

[Cap'n Cat]

Every time elections come around I have to shake my head over the way pundits look at modest differences in day to day polls then launch into analyses of why things changed. Fact is, a lot of the time, the polls don't provide sufficient evidence to say anything changed.

For purposes of illustration, I ran a simulation in which the true percent support in the population was 40% for Candidate A and 40% for Candidate B with 20 percent undecided. I did 10 runs of random samples of 1000 each. This is like taking 10 political polls consecutively to represent the same situation.

Among the 10 runs the "best" poll result I got for Candidate A was A getting 41.0% to B getting 36.6% with 22.4% undecided. The "best" poll result for Candidate B was B getting B getting 42.6% to A getting 36.9% with 20.5% undecided. So you're talking about a 10.1percentage point swing between the two with A being ahead by 4.4% in one poll and B being ahead by 5.7% in the other.

But the reality is that there was NO DIFFERENCE between the actual percentages in the population. The difference in the simulated poll results is just random sampling error.

It is SO ridiculous that people see changes in these polls and start acting as though that means there necessarily was an actual change in the true percent support for each candidate in the population. But I guess it gives us something to watch on TV and talk about.

See me in mid-November, St. Wronge!



GObama!!!

[JohnStOnge]

See$ me in mid-November, St. Wronge!



GObama!!!

Cap, I've said since Obama got elected that I think he'll get re-elected. That's because he came in when the economy was in a swoon. That, to me, meant that he'd get cut a lot of slack plus the odds were that things would get better just in the "natural" cyclical course of things. And now we know that the Republicans don't have any "charisma" candidates so it looks even better for him. I'll be shocked if he doesn't win.

What I'm talking about has nothing to do with that. I'm just talking about how oblivious people appear to be with respect to the fact that even a perfectly executed sample survey does not give one a measurement of reality. It gives an ESTIMATE of reality.

Like last night I was talking to my wife about approval ratings. I noted that if Obama had a true approval rate of 47% and we planned to take a random sample of 1000 to estimate it, the 95% interval for the estimate would be from 43.9% to 50.1%. So getting 44% and getting a 50% would both be plausible given the truth being 47%. But if one month Obama gets a 50% estimate and the next month he gets 44% the pundits think he's in free fall when the truth could be that he's about in the same place or very close.

The part that concerns me is that the pundits' inability to correctly interpret sample estimates may actually influence the reality. Like for instance in a primary it might make voters looking to ride the winning horse misjudge what the actual situation is.

[Cap'n Cat]

See$ me in mid-November, St. Wronge!



GObama!!!

Cap, I've said since Obama got elected that I think he'll get re-elected. That's because he came in when the economy was in a swoon. That, to me, meant that he'd get cut a lot of slack plus the odds were that things would get better just in the "natural" cyclical course of things. And now we know that the Republicans don't have any "charisma" candidates so it looks eve better for him. I'll be shocked if he doesn't win.

What I'm talking about has nothing to do with that. I'm just talking about how oblivious people appear to be with respect to the fact that even a perfectly executed sample survey does not give one a measurement of reality. It gives an ESTIMATE of reality.

Like last night I was talking to my wife about approval ratings. I noted that if Obama had a true approval rate of 47% and we planned to take a random sample of 1000 to estimate it, the 95% interval for the estimate would be from 43.9% to 50.1%. So getting 44% and getting a 50% would both be plausible given the truth being 47%. But if one month Obama gets a 50% estimate and the next month he gets 44% the pundits think he's in free fall when the truth could be that he's about in the same place or very close.

The part that concerns me is that the pundits' inability to correctly interpret sample estimates may actually influence the reality. Like for instance in a primary it might make voters looking to ride the winning horse misjudge what the actual situation is.

I understand what you're saying, J, and agree. But, it's too much for fun for me to rub polls into Conk faces. They are so high and mighty.