Silver 10/18 Nate is head of the NYT’s 538 blog. “Gallup vs. the World,” 2012, http://fivethirtyeight.blogs.nytimes.com/2012/10/18/gallup-vs-the-world/?gwh=5E00DF13001CF76F90FE57382BE66AC8
The Gallup national tracking poll now shows a very strong lead for Mitt Romney. As of Wednesday, he was ahead by six points among likely voters. Mr. Romney’s advantage grew further, to seven points, when Gallup updated its numbers on Thursday afternoon.¶ The Gallup poll is accounted for in the forecast model, along with all other state and national surveys.¶ However, its results are deeply inconsistent with the results that other polling firms are showing in the presidential race, and the Gallup poll has a history of performing very poorly when that is the case.¶ Other national polls show a race that is roughly tied on average, while state polls continue to indicate a narrow advantage of about two points for President Obama in tipping-point states like Ohio. The forecast has Mr. Obama as a narrow favorite in the election largely on the basis of the state polls. (You can read my thoughts here on the challenge of reconciling state and national poll data.)¶ Our database contains records from 136 distinct pollsters that have released at least one state or national survey at some point in this election cycle. Of those, 53 are active enough to have issued at least one survey since Oct. 1.¶ With so much data to sort through, it will usually be a counterproductive use of one’s time to get overly attached to the results of any one particular poll. Whether you look at the relatively simple averaging methods used by Web sites like Real Clear Politics, or the more involved techniques in the FiveThirtyEight forecast, the Gallup national tracking poll constitutes a relatively small part of the polling landscape.¶ Let me walk you through the rules for how the FiveThirtyEight model weighs the Gallup poll relative to all the other information it accounts for. This explanation will be modestly technical — you may want to skip ahead to the next section if you aren’t concerned with these details.¶ The Role of the Gallup Poll in the FiveThirtyEight Model¶ There are two major pieces of information that we’re looking to extract from each poll. One is simply the raw number — who is ahead or behind? The other is the trend it shows in the race — which candidate is gaining or losing ground?¶ Different types of polls are relatively more and relatively less useful for these purposes. Because national tracking polls like Gallup are published every day, they are useful for the trend part of the calculation, measuring the change in the race against a constant baseline.¶ Each poll receives a weight in the FiveThirtyEight trend-line calculation based on its sample size and its pollster rating. The model accounts for the fact that tracking polls use an overlapping set of interviews. A three-day tracking poll might consist of interviews conducted on Monday, Tuesday and Wednesday, for instance. When the polling firm issues its next release of the survey, a fresh set of interviews from Thursday will replace the ones from Monday in the sample. Thus, we reduce the weight assigned to each edition of a tracking poll to avoid counting the same interviews multiple times.¶ Even so, there are quite a few interviews conducted by a tracking poll over the course of a week — about 3,000 per week in the Gallup national tracking poll, for instance.¶ But Gallup is not the only national tracking poll. There are six published on most days; the others are from Rasmussen Reports, Ipsos, the RAND Corporation, Investors’ Business Daily and United Press International. (A seventh daily tracking poll, from Public Policy Polling, made its debut on Thursday.)¶ Of the daily tracking polls, the Gallup survey receives the largest weight in the trend-line calculation. It uses a larger sample size than most other polls, and it has a methodology that includes calls to cellphone voters.¶ On the other hand, the pollster ratings are also based in part on past accuracy, and Gallup’s performance is middling in that department. It mostly gets a lot of weight by comparison, since the tracking surveys are a mediocre group on the whole.¶ The trend-line adjustment also looks at other national polls when they are published, like the New York Times/CBS News or the Wall Street Journal/NBC News surveys. This is a high-quality group of polls; the disadvantage is that they are published only occasionally.¶ State polls are also useful for determining the overall trend in the race. In this case, the advantage is the abundance and diversity of data: there might be 10 or 20 state polls published on a typical day, often from 5 or 10 polling firms.¶ The trend-line calculation applies a 50 percent penalty to the weight assigned to state polls because trends in any one state could be an aberration. However, the states generally rise and fall together when there is a shift in the national climate — and if one candidate makes an especially large gain in one state, it must necessarily be counterbalanced by others in which it is below average.¶ The relative amount of weight assigned to each type of poll is fluid rather than fixed, and depends on the overall volume of data. On days when a large number of state polls is published but few national ones, they will generally be the more useful source for making inferences about the trend in the race.¶ But on average since Oct. 1, the Gallup national tracking poll has accounted for 12 percent of the information that the model uses to calculate the trend line. The other daily tracking polls, collectively, have accounted for 24 percent of the data, and the occasionally published national polls for 19 percent. Finally, the state polls account for about 45 percent of the data used to calculate the trend-line adjustment.¶ Thus, even though the Gallup national tracking poll is more influential than any other individual poll series in the FiveThirtyEight trend-line calculation, it still accounts for only about 12 percent of it. It can very easily be outweighed by the other polls if they are in disagreement with it.¶ As I mentioned, however, this is only half the battle. Once “old” polls are brought up to date by adjusting them to reflect the current trend in the race, we still need to take some kind of average of them.¶ The way the polls are employed to calculate the average is a little different than in calculating the trend line. Our research suggests, for instance, that state polls, rather than national polls, often provide a better estimate of the national popular vote, in addition to the Electoral College.¶ In addition, although the trend-line calculation relies fairly heavily on the quantity of polling — even a mediocre poll can be useful for measuring how the race is changing if it is published frequently — the polling average tends to place more emphasis on the quality of the poll. (Otherwise high-quality polls that are a bit of out-of-date can still have a fair amount of influence on the average. The problem with these polls being less recent is mitigated because the trend-line adjustment serves to make them more contemporary in the event that there has been a significant shift in the race.)¶ Over all, the Gallup daily tracking poll accounts for only about 3 percent of the weight in this stage of the calculation. The national tracking polls collectively, including Gallup, account for only about 10 percent of it. Most of the weight, instead, is given to the state polls.¶ This is, obviously, a rather detailed answer to the seemingly simple question of how much information is provided by the Gallup national tracking poll, as opposed to all the other state and national surveys.¶ Nevertheless, any rigorous attempt to consider the value of the Gallup poll would probably get you to something of the same answer. Perhaps the Gallup poll accounts for 5 or 10 percent of the information that an election analyst should evaluate on a given day.¶ The Gallup poll’s influence on the subjective perception about where the presidential race stands seems to be proportionately much greater than that, however — especially when the poll seems to diverge from the consensus.¶ This simply isn’t rational, in my view. As I discuss in my book, our first instincts are often quite poor when it comes to weighing information. We tend to put too much emphasis on the newest, most widely reported and most dramatic pieces of data — more than is usually warranted.¶ Gallup Performs Poorly When Out of Consensus¶Usually, when a poll is an outlierrelative to the consensus, its results turn out badly.¶You do not need to look any further than Gallup’s track record over the past two election cycles to find a demonstration of this.¶In 2008, the Gallup poll put Mr. Obama 11 points ahead of John McCain on the eve of that November’s election.¶That was tied for Mr. Obama’s largest projected margin of victory among any of the 15 or so national polls that were released just in advance of the election. The average of polls put Mr. Obama up by about seven points.¶The average did a good job; Mr. Obama won the popular vote by seven points. The Gallup poll had a four-point miss, however.¶In 2010, Gallup put Republicans ahead by 15 points on the national Congressional ballot, higher than other polling firms, which put Republicans an average of eight or nine points ahead instead.¶In fact, Republicans won the popular vote for the United States House by about seven percentage points — fairly close to the average of polls, but representing another big miss for Gallup.¶ Apart from Gallup’s final poll not having been especially accurate in recent years, it has often been a wild ride to get there. Their polls, for whatever reason, have often found implausibly large swings in the race.¶ In 2000, for example, Gallup had George W. Bush 16 points ahead among likely voters in polling it conducted in early August. By Sept. 20, about six weeks later, they had Al Gore up by 10 points instead: a 26-point swing toward Mr. Gore over the course of a month and a half. No other polling firm showed a swing remotely that large.¶ Then in October 2000, Gallup showed a 14-point swing toward Mr. Bush over the course of a few days, and had him ahead by 13 points on Oct. 27 — just 10 days before an election that ended in a virtual tie.¶ In 1996, Gallup had Bill Clinton’s margin over Bob Dole increasing to 25 points from nine points over the course of four days.¶ After the Republican convention in 2008, Gallup had John McCain leading Mr. Obama by as many as 10 points among likely voters. Although some other polls also had Mr. McCain pulling ahead in the race, no other polling firm ever gave him larger than a four-point lead.¶ It’s not clear what causes such large swings, although Gallup’s likely voter model may have something to do with it.¶ Even its registered voter numbers can be volatile, however. In early September of this year, after the Democratic convention, Gallup had Mr. Obama’s lead among registered voters going from seven points to zero points over the course of a week — and then reverting to six points just as quickly. Most other polling firms showed a roughly steady race during this time period.¶ Because Gallup’s polls usually take large sample sizes, statistical variance alone probably cannot account these sorts of shifts. It seems to be an endemic issue with their methodology.¶ To be clear, I would not recommend that you literally just disregard the Gallup poll. You should consider it — but consider it in context.¶ The context is that its most recent results differ substantially from the dozens of other state and national polls about the campaign. It’s much more likely that Gallup is wrong and everyone else is right than the other way around.