Science of polling is really a dark art

Published 10:41 pm Friday, September 20, 2013

A recent debacle in political polling serves as a great case study. The lesson to be learned here is that polling isn’t a science, it’s an art — and a dark art, at that. While much of American political reporting focuses on polls, they should never be accepted at face value, and never substituted for actual news.

The debacle happened in Colorado, in the days leading up to an historical recall election. The nationally recognized polling agency Public Policy Polling (PPP) conducted a survey the weekend before the vote, and then promptly sat on the results.

Senator Angela Giron and Senate President John Morse are moderate Democrats who supported gun control. They were the subject of the state’s first-ever recall election on Sept. 10.

“PPP found 54 percent of Pueblo’s registered voters overall favored Giron’s recall, while just 42 percent opposed it,” Pacific Standard magazine reports. “Stranger yet, a whopping 33 percent of Democratic respondents said that they supported the recall. Consider that 47 percent of Pueblo’s registered voters are Democrats; just 23 percent are Republicans.”

PPP pollsters didn’t know what to make of the results.

“In a district that Barack Obama won by almost 20 points I figured there was no way that could be right,” said PPP Director Tom Jensen.

But the election results matched the poll numbers. In fact, the recall passed 56-44, slightly higher than even the poll showed.

Conservatives and gun rights proponents criticized the polling agency. Even the nation’s most respected pollster, Nate Silver of the fivethirtyeight blog, chimed in.

“VERY bad and unscientific practice for PPP to suppress a polling result they didn’t believe/didn’t like,” Silver tweeted.

That’s missing the point. PPP was right to sit on the results, because they had good reason to doubt them. It could have been a design flaw — a poorly worded question, for example. That’s what PPP suspected.

“This was the first legislative recall election in Colorado history,” Jensen said. “There’s been a lot of voter confusion. That finding made me think that respondents may not have understood what they were being asked.”

By quashing the poll, PPP was acknowledging that it’s not an exact science. Polls are highly dependent on subjective factors, such as how questions are worded, which demographics are represented in the polling sample, and even who was home at the time the pollsters called.

“It’s refreshing to find that there are still pollsters who self-regulate,” Pacific Standard’s Rebecca Buckwalter-Poza says. “Too many pollsters operating in the political sphere take an Ivory Tower attitude, disavowing responsibility for the consequences of their polls and analyses.”

Polls can have an effect on the outcome of an election; so-called “push polls” are designed to influence those being questioned and they’re a time-dishonored campaign tactic.

But even polls that strive to be “fair” can be flawed. No poll must ever be taken at face value, because there are so many subjective factors. Political reporters must look deep into the numbers. Otherwise, a poll is just a distraction from the real issue at hand.