Polling And Election 2016: What The Heck Happened? An Interview With Pollster Charles Franklin

Well, that was… surprising. Pre-election predictions, based on polls from around the country, gave Hilary Clinton a solid chance of beating Donald Trump in the presidential election — north of 60 percent on Five Thirty Eight, and north of 80 percent on the New York Times. And yet here we are, on the morning after Election Day, with Donald J. Trump as our president-elect. What the heck happened?

Early this morning, after about two hours of sleep, I found myself at Newark Airport, on my way up to SUNY Buffalo Law School to speak to the school’s Federalist Society chapter (at 3 p.m. in 107 O’Brian Hall, for anyone who’s interested). And who did I have the good fortune to bump into at my gate? Professor Charles Franklin of Marquette Law School — one of the nation’s leading pollsters, Director of the Marquette Law School Poll, and founder of Polls and Votes, a site focused on non-partisan analysis of the details of polling and election outcomes.

Franklin was returning to Milwaukee from New York after serving on the “decision desk” of ABC News — the pollsters, statisticians, and analysts who break down election returns and help the network call different states. So of course I had to ask him: what the heck happened here, in terms of pre-election polls missing the mark by so much?

(Please note that Franklin’s views are his own, not those of Marquette Law or ABC News. Also, he couldn’t discuss the specifics of his work for ABC — other than to reveal that he was at the decision desk until 4 a.m., at which point he went to the airport. He seemed quite alert and animated despite having basically no sleep.)

Franklin opened with some self-deprecating humor: “I got everything wrong, but at least I’m in good company!” Pretty much every major polling organization and pollster failed to predict Trump’s victory, he noted.

This was, according to Franklin, the worst election for polling since the 2015 elections in the United Kingdom, in which pollsters failed to predict the conservative majority. Franklin said that election is a better comparison to last night than the Brexit vote — which gets mentioned more, in part because Trump made much ado about the Brexit polls and outcome — because the Brexit polls actually predicted a close vote in which either side might prevail.

“It will take some time before we have any kind of serious analysis of where things went wrong,” Franklin said of election 2016. But he did offer some preliminary thoughts.

There are two key questions, Franklin explained: (1) whether pollsters represented groups correctly, i.e., did they have the right percentages of different demographics in their samples, and (2) whether pollsters failed to predict how members of the different groups would vote.

For example, on gender, did the samples used by pollsters contain the same proportion of men and women as the actual electorate? If so, then did the groups just not vote the way they were expected to? Taking his home state of Wisconsin, Franklin observed that pollsters got the gender balance basically correct — they expected 52 percent women and 48 percent men, and exit polls showed 51 percent women and 49 percent men in the actual electorate. But pollsters failed to detect how men went for Trump in considerably higher numbers than pre-election polling suggested.

Another example: partisanship, or the representation in the sample of Republicans, Democrats, and independents. On this front, again focused on Wisconsin, independents supported Trump much more strongly compared to where pre-election polls had them. Pollsters noticed some shift among Wisconsin independents in the past six weeks, but nothing near the size of what the exit polls ultimately showed. Much of this was late-breaking: around 56 to 58 percent of independents voted for Trump if they made up their mind in the last week. So this does raise the possibility that FBI Director Jim Comey’s last-minute communications about the investigation into Hillary Clinton’s emails affected votes, although it’s hard to nail down how much of an effect this had (at least until we get information from additional sources, discussed below).

I asked Franklin about one frequently mentioned theory that Trump voters didn’t want to admit to pollsters their support for the controversial candidate. This concern was actually on the radar of pollsters, he noted. Morning Consult conducted a randomized experiment in which half of poll respondents expressed their opinions to live surveyors and the other half gave their views to a machine, and it found no difference, at least right before the election. There was a difference when this experiment was conducted in late 2015 — automated or machine-conducted polls showed more support for Trump, to the tune of around four points — but that difference appeared to go away in the spring 2016 primary season. While it’s hard to prove a negative, pollsters did make an honest attempt to evaluate this effect, and they didn’t see it.

(One possibility that this experiment would not catch, Franklin acknowledged: Trump voters simply refusing to participate in polls altogether — the so-called “hidden Trump voter.” But he added that polls did not systematically underestimate the Trump vote during the primaries, across multiple elections in different states. So it’s not clear why the general election should be different. Of course, once again it’s hard to prove or disprove this theory, because by definition these hidden voters didn’t talk to pollsters.)

When can we expect more definite answers of what happened here? Not for months, Franklin explained, as we get better information about the composition of the November 2016 electorate. That information will come from three main sources: (1) exit polls, (2) the Current Population Survey (CPS) of the Census Bureau, and (3) voter lists. Right now we just have exit polls; in the weeks and months ahead, we’ll get the other two sources.

Exit polls, as their name suggest, involve voters being approached in person as they exit polling places and asked to fill out a questionnaire. The questionnaire asks them for information about who they voted for; demographic data, such as age, race, and gender; and why they voted the way they did (e.g., questions about how much they trust the candidates, or their views on the candidates’ qualifications). Exit polls aren’t perfect — they’re self-reported, so respondents might lie, and not everyone participates, such as voters in a hurry — but they are still important for figuring out the composition of the electorate in a given cycle. (This is their best use, according to Franklin, even though news organizations also rely upon them for forecasting outcomes on Election Day.)

The Current Population Survey is conducted monthly; it’s the basis for monthly unemployment figures. In November of each election year, it includes a voter supplement asking respondents if they are registered voters and whether they did in fact vote in the most recent election. The CPS also contains a wealth of demographic data — age, sex, race, marital status, income, and more — and so it is an excellent source for figuring out the composition of the electorate in a given cycle. It involves a large sample size, more than 100,000 people, and boasts a high response rate, around 90 percent (compared to the typical eight or nine percent for a typical phone survey). We won’t get the November CPS for a few weeks.

Finally, there are the voter lists, which provide information about whether a given individual voted in this particular election (but again, not who the person voted for). The voter lists are important because exit polls and the CPS do not track individual voters, while the voter lists do. The lists take data acquired from election authorities about a given individual’s registration and voting history and then supplement it with demographic information acquired from data brokers. Compiling and maintaining these lists takes money — some election authorities require payment for access to voter rolls, and data brokers definitely require payment — and so they tend to be put together by political parties and their allies.

It will take weeks or even months to get voter lists fully updated. States take time to certify official counts, which for some jurisdictions won’t happen until mid-December, and then outside organizations will have to obtain that information from the election authorities. But once the voter lists are updated, they will offer powerful insights into who exactly voted in this election, allowing pollsters to better determine what factors or groups drove Donald Trump to victory. What role was played by first-time voters? Or voters who sat out the past few elections and returned? Was there a surge of turnout in Republicans, or a dip in turnout of Democratics? Voter lists, with their granular data about individual citizens, can enlighten us — and give pollsters a better sense of where they went wrong.

I concluded by asking Franklin: what effect will election 2016 have upon public perception of polling?

“The results last night clearly show that pollsters as a group can make systematically incorrect conclusions,” Franklin said. “This will surely add to skepticism about polling and the ability to sow doubt about polls.”

“In 2012, Mitt Romney’s campaign pushed the idea that the polls were all wrong, and polling was vindicated. But this year, the shoe wound up on the other foot.”

---

David Lat is the founder and managing editor of Above the Law and the author of Supreme Ambitions: A Novel. You can connect with David on Twitter (@DavidLat), LinkedIn, and Facebook, and you can reach him by email at [email protected].