Debating the Data
“Stick to the facts,” have you ever had someone tell you that? To just stick to the facts as you tell a story or communicate a position? Because, as another saying goes, “the facts don’t lie.”
So what happens when the facts seem to contradict each other? When the data doesn’t all line up? Are the facts lying after all? Or is it just our inability to accurately interpret them?
I was recently asked by Jonathan Merritt, a columnist at The Atlantic, to comment on the idea of bias in polling—whether certain social expectations can cause survey participants to answer questions with the answer they believe the researcher is looking for. It’s called “social desirability bias” and it’s been pointed to by a few people as a possible cause for the relatively quick shift in public opinion toward support of same-sex marriage. Merritt wanted to know if I thought there was any validity to that argument—or if another bias was at work here: confirmation bias. Were the pundits eager to accept the facts that supported their beliefs but quick to question or even malign the facts that stood in opposition to them?
In these conversations with Merritt, two observations struck me:
First, every survey finding is subject to the language and context in which it was asked. On topics that generate a great deal of cultural conversation—for example, same-sex marriage and the swift shift in public opinion—it is easy for one set of polling to misread the scope of what is happening in public opinion. But almost all the polling points to the fact that attitudes on same-sex marriage and LGBT issues are changing. What is up for debate is how deep those shifts are and why they are happening.
For example, in one of our polls after the SCOTUS decision, we found that 62% of Americans felt the legalization of same-sex marriage was inevitable. Among evangelicals, only 2% supported the actual decision, but 31% said the decision was inevitable.
Another example: 54% of Americans and 24% of evangelicals believe Christians can support legal same-sex marriage and yet also affirm the church’s traditional definition of marriage between one man and one woman.
So, this is showing that people—across the faith spectrum, but especially evangelicals—are trying to find their footing on this topic. And I think it shows that, when it comes to poll questions, it’s possible that some people could respond by saying they support gay rights, but they might also be feeling that it’s just the thing to do. It’s inevitable.
Second, the conversation with Merritt and the surrounding debate about the polls, reminded me of the controversy that many statistics generate. In my two decades of work in research, I’ve seen all sorts of examples of people challenging stats they don’t agree with. The psychology of how people use and apply stats is really interesting—how confirmation bias actually is a real thing.
It seems that data love debate. In most arenas of culture—media, the economy, retailing, healthcare, government, and so on—there is a debate about what is really true. The same thing happens with Christian statistics: there are different sources of information about the world of faith, but when these sources conflict, we often resort to impugning the motives and methods of others.
When people agree with research findings, they rarely or never question the methodology. However, when people don’t agree with data findings, they either just ignore it or find reasons to gripe about the methods and questions asked.
Good research seems to have a mix of things that both fit and don’t correspond to our presuppositions. That’s why we do research—to confirm or repudiate our assumptions as humans.
I believe we should have good, healthy debate about data—their accuracy and meaning. The more important the decisions we are making, the more crucial it is that we get our data right. But, from my standpoint, in the Christian community we too often resort to the wrong spirit of “myth busting” on the work of our brothers and sisters.
I predict that the debate over data will increase in the next decade. There are more of us doing research about Christianity and faith. The threshold to enter the “research” field is as low as ever (hello, Survey Monkey). And the world is changing very quickly, so we need insight to make sense of the change.
My suggestion: we need to work very hard at finding constructive, Jesus-like ways of debating data.