Can We Trust the Polls?

The AAPOR committee chased every popular theory for what went wrong with polling in those states — including the “shy Trump voter” effect, the belief that some of his voters were reluctant to reveal their intention. For that one, it found no significant evidence.

What the committee did find was an unprecedented change in voter preference late in the campaign. Roughly 15 percent of voters in Wisconsin, Michigan, and Pennsylvania waited to make a decision until the final week of the race, when most polls were no longer actively in the field.

“Normally, those late deciders aren’t much of a factor,” Kennedy says. “Historically, they break about evenly between the two major-party candidates, so it kind of washes out. In 2016, in those battleground states, they broke for Trump by wide margins — 15 to 20 percent.”

Such a divergence would have been difficult for pollsters to anticipate, but the committee did identify a significant failing. Many polls, especially at the state level, neglected to adjust for an overrepresentation of college graduates in their samples. And those voters turned out strongly for Clinton.

“People with higher levels of formal education tend to be more likely to take polls,” Kennedy says. “And we’ve known this, frankly, for a long time.”

In elections past, polls could get away with not weighting for education, assuming they accurately adjusted for more predictive demographics like gender, age, and race. But that’s because Democrats successfully courted some white working-class voters. Not long ago, especially among union workers, they were a key constituency of the Democratic base. “And in 2016, Trump turned that on its head,” Kennedy says.

In a way, election polling’s reckoning was inevitable. Local news organizations have historically led or cosponsored polling efforts. And the quantity and quality of state-level polling has followed the trajectory of declining media budgets.

“It is a persistent frustration within polling and the larger survey research community that the profession is judged based on how these often-underbudgeted state polls perform relative to the election outcome,” the AAPOR report concluded. “The industry cannot realistically change how it is judged, but it can make an improvement to the polling landscape.”

And UW–Madison answered the call.

The Battleground Poll

“Wisconsin is the quintessential battleground state,” says Barry Burden, director of the UW’s Elections Research Center and professor of political science. “Almost magically, the forces that would help Republicans and the forces that would help Democrats seem to be in a perpetual balance.”

Most notably, while urban areas like Madison and Milwaukee are turning out strongly for Democrats, rural voters in the northern and western parts of the state are gravitating to Republicans.

And yet, aside from the highly regarded Marquette poll, the state lacks consistent public-opinion surveys to capture such trends.

UW–Madison was uniquely positioned to fill this polling gap. In 2015, the Department of Political Science established the Elections Research Center to centralize the research of more than a dozen faculty members across disciplines.

Many of the center’s experts have earned national reputations, appearing regularly in the New York Times, the Wall Street Journal, PolitiFact, and other media. Burden alone fields as many as 10 requests from journalists per day, with recent topics ranging from caucus-reporting technology, to diversity on the presidential ticket, to mail-in voting, to the electoral consequences of a pandemic.

The experts are called to testify in court and legislative bodies about their research. The Washington State Senate, for instance, asked Burden to share his findings on the effects of same-day voter registration, which the center had studied in Wisconsin.

To reach the broader public, they post to Twitter during debates, election nights, and other major political events. “[The work] requires a level of vigilance to keep up with current events,” Burden says. “One of the things my colleagues and I can offer is putting these events in context of research and broader patterns.”

The Elections Research Center partners with the data company YouGov and media entities to conduct public-opinion surveys. One survey made headlines during the 2018 Senate hearing on sexual assault allegations against Supreme Court nominee Brett Kavanaugh. UW researchers recycled questions from a survey in 1991, when Justice Clarence Thomas faced sexual-misconduct allegations. They found almost no movement in public opinion on whether the respective nominations should go through if the allegations were true, with around 70 percent of respondents in both surveys saying it should not. (However, respondents in 2018 indicated they were much more likely to vote against a senator if he or she supported the nominee.)

Encouraged by past efforts, the center launched the 2020 battleground poll, immediately contributing to the political discussion. Its first poll results in February dispelled a pervasive cable-news narrative. Throughout the 2020 Democratic presidential primary, the networks used exit-polling data to demonstrate that the top motivation for Democratic voters was “electability” — whoever had the best chance to beat Trump.

Burden says surveys can shed light on an election beyond the question of who voted for whom. Jeff Miller

But Burden wondered whether that was the result of Democratic voters being faced with a false dichotomy: do you care about the issues or beating Trump? When the UW’s poll offered more options to respondents, it found that 37 percent predominantly supported a primary candidate because of key issues, 22 percent because of his or her chances to beat Trump, and 20 percent because he or she was the most qualified to be president.

And to no one’s surprise: the poll also showed a close race between Trump and several of the then–Democratic candidates in Wisconsin, Pennsylvania, and Michigan. The center plans to conduct several more surveys before November’s election.

The battleground poll continues a legacy of elections research at the university, though conditions have vastly changed over the past two decades.

The Landline Era

From 1998 to 2008, UW political scientists ran the Wisconsin Advertising Project, which analyzed presidential and statewide campaign ads aired on TV in more than 200 media markets. It pulled back the curtain on how candidates communicated with voters.

Around the same time, the UW Survey Center administered the Badger Poll to measure statewide public opinion. Cramer, who’s now an affiliate of the Elections Research Center, served as the faculty director of the Badger Poll. She started to notice overlapping trends in polling results and her fieldwork. “Many people thought that rural areas of the state didn’t get their fair share of state taxpayer dollars,” she says.

The Badger Poll’s methodology was to sample some 500 Wisconsin residents by randomly selecting households with active landlines. It mailed notices to households, redialed numbers as many as 10 times, and intensively trained interviewers to keep potential respondents on the line. Those efforts helped the Badger Poll, which ran 32 surveys between 2002 and 2011, regularly achieve a response rate of roughly 40 percent.

Now, such a response is a fantasy. If you’re willing to answer a call from an unknown number, you’re part of the 1 to 2 percent response rate that even high-quality phone surveys strive for today.

For as long as polling has existed, technology has uprooted methodology. And with the public becoming more difficult — and costly — to reach, pollsters are getting creative. The UW’s polling partner, YouGov, is an entirely online operation that’s finding ways to forgo representative sampling without sacrificing quality.

The Brave New World of Polling

By 1936, the Literary Digest had accurately predicted the winner of every presidential election of the past 20 years. That year, it embarked on one of the most ambitious polling operations of all time, mailing out mock ballots to 10 million people.

“When the last figure has been totted and checked, if past experience is a criterion, the country will know within a fraction of 1 percent the actual popular vote,” the Digest declared.

The magazine’s straw poll predicted that Franklin D. Roosevelt would lose in a landslide to Alf Landon and win just 43 percent of the vote. He won 61 percent.

That cataclysmic error — the result of selection bias with a disproportionately affluent mailing list as well as a high rate of nonresponses — ushered in the modern era of scientific public-opinion polling. George Gallup and Elmo Roper, with much smaller but more representative samples, both predicted Roosevelt’s reelection and went on to found polling enterprises that remain active today. (The Digest folded 18 months later.)

For some 50 years, representative sampling easily differentiated the good polls from the bad. But as communications technology has evolved, the craft of high-quality polling has become much more complex. Whenever a new, pricier technology emerges — landlines, cellphones, the internet — polling must adapt to it slowly or risk skewed samples because of lack of widespread access. The internet has posed another problem: while pollsters can access lists of every mailing address or phone number, there’s no master list of emails, making it impossible to recruit a representative sample digitally.

The UW’s polling uses an opt-in panel of people whom YouGov has recruited online and compensated for participation in other surveys (such as product testing for companies). While it’s not representative sampling, YouGov collects a wide range of personal information from its participants, which it uses to accurately weight the sample to the larger population. YouGov’s polls have called nearly 90 percent of recent political races correctly, according to FiveThirtyEight.

YouGov has learned to innovatively adjust for variables beyond demographics, including political ideology, volunteerism, and news consumption. Such measures have helped it address polling’s familiar foe: the overrepresentation of people with college degrees.

“This is not a fly-by-night internet operation,” Burden says. “They are staffed by a lot of academics and political scientists who understand the scientific goals we have.”

High-quality polls seem to be navigating this new landscape effectively. The 2017–19 election cycle was one of the most accurate on record for polling, according to FiveThirtyEight, with the lowest average error margins since 2003–04.

So where does that leave us for 2020?

Trust Polling to Do What?

As I talked to the experts in this story, I started with the same question: “Why is polling important?”

In March, I read Jill Lepore’s New Yorker article “The Problems Inherent in Political Polling.” She argued that polling, like endeavoring to gauge public opinion via social media, is flawed. “Democracy requires participation, deliberation, representation, and leadership — the actual things, not their simulation,” she wrote.

I, too, had become a skeptic.

But the experts reminded me that — beyond the horse-race polling numbers that tend to populate the headlines — collecting public opinion can serve a critical function of democracy.

“If you think about a society where there’s no polling, the leader of that country can just go out and say, ‘Well, I know the public feels this way,’ ” Kennedy says. “Contrast that with a society where there are a lot of independent pollsters and all of their data converge, so that, ‘Oh, actually, the public has this other attitude.’ It provides a reality check for people in office.”

According to Burden, surveys can shed light on an election beyond the question of who voted for whom. “Public opinion polling asks voters directly, ‘What are you thinking? What matters to you? What are your positions on the issues? What would you like to see government do? What is your ideal vision of society?’ ”

I had also lost sight of the golden rule of polling: that the results represent a snapshot in time, not a prediction. For pollsters, it’s a clichéd disclaimer. But for a public and media eager to speculate on who will win, misinterpreting polls as predictions can lead to false expectations and big surprises.

Through high-quality polling, insightful public-opinion analysis, and research-based commentary, UW experts are demonstrating a responsible way forward for our political dialogue. They’re uncovering how voters feel, not projecting what they will do. Cramer discovered rural resentment the old-fashioned way: “polling by walking around.”

As for the question everyone is asking — “Should I trust the polls after 2016?” — I saved that one for last.

“My response is, ‘Trust polling to do what?’ ” Kennedy says. “Are you asking me if you can trust polling to call the winner in a close election? I would say no. It’s not up to that task. But if the question is, ‘Can I trust polling to tell me how the public feels about Donald Trump? About [House Speaker] Nancy Pelosi? About health care? About the response to the COVID-19 pandemic?’

“Yes, polling is absolutely up to that challenge.”