Reading religion surveys can seem like confronting the Tower of Babel: stacked questions, confusing terms, unscientific methodology.
It gets even crazier when results are contradictory. How does that happen?
Some surveys lean like the Tower of Pisa
The Pledge of Allegiance is a perfect example.
There’s almost always a flap over how many Americans do — or don’t — want the words “under God” kicked out of the Pledge of Allegiance. Indeed, on Nov. 19 a court in Monmouth, N.J., will hear the case of the American Humanist Association battling the Matawan-Aberdeen Regional School District to have schools edit out mention of God.
The humanists claim 34 percent of Americans agree with their view. But, wait. What about a survey conducted earlier this year by LifeWay Research, a Christian research agency? It found that only 8 percent would cut God from the Pledge.
Why four times the difference? Look to the poll language.
LifeWay asked: “Should the words ‘under God’ be removed from or remain in the Pledge of Allegiance to the United States of America?” That’s a straight-up question with no preface.
The humanists’ survey, however, began with a bit of pointed Pledge history — before getting to the (loaded) question:
“For its first 62 years, the Pledge of Allegiance did not include the phrase ‘under God.’ During the Cold War, in 1954, the phrase ‘one nation, indivisible … ‘ was changed to read ‘one nation, under God, indivisible … ‘. Some people feel this phrase in our national pledge should focus on unity rather than religion.
“Do you believe the Pledge of Allegiance should:
- Return to the unchanged version: ‘one nation, indivisible … ‘
- Continue with the changed version: ‘one nation, under God, indivisible … ‘”
This is not kosher poll methodology, say experts.
“Always ask yourself why this group sponsored this survey,” advised David Kinnaman, president of the Christian research company Barna Group. “Read the questions and see if the responses are prompted. What is the information asking me to fear or to love? Are they trying to elicit one of those emotions from me?”
Watch the labels
Researchers on religion and politics are fascinated with the evangelical vote. Is it growing? Shrinking? Trending X or Y direction?
But “evangelical” is one of the slipperiest words out there. Since every survey group sets its own definition, results can confuse more than they enlighten.
This has long been true. In 1998, Gallup asked people if they were “evangelical or born again” and came up with 47 percent, says survey research veteran Conrad Hackett, a demographer at the Pew Research Center’s Religion & Public Life Project.
But University of Notre Dame sociologist Christian Smith came up with 7 percent that same year. Smith counted as evangelical those who said they were “active in a congregation, and they had to be Protestant, and they had to say their faith is important to them,” said Hackett.
Pinning down religious identity in an era when 20 percent are “nones” — people who say they have no particular faith brand — is like trying to climb a greased pole. Neither can you assume that a label reflects reality, or that identity, belief, and actions align. Oy!
Speaking of “Oy”… Pew Research dealt with the complex question of “who is a Jew” by allowing people to define themselves by religion, culture, and family ancestry. However, there’s no parallel spot for cultural Christians who have little or no commitment to Christian theology or religious practice.
No survey has a “Christian Lite” category. Maybe they should. When Kinnaman at Barna Group looked at the mix of belief and behavior and church involvement, his organization’s survey found that 38 percent of self-labeled Christians were essentially like nones in their political and cultural outlook.
People fib
Raise your hand if the weight listed on your driver’s license is correct — or ever was. Well, we fib on surveys, too.
When Philip Brenner, a University of Michigan research fellow with the Institute for Social Research, examined hundreds of surveys and time diaries, he found Americans over-report their church attendance by 10 percent to 18 percent.
Why? We give answers that fit our self-image, Brenner said. We reframe the question to be: “Are you the sort of person who attends religious services?” Sure we are.
Methodological madness
We’re all bombarded with online opportunities to answer surveys. Fun — and totally unscientific. Put no credence in the results because they’re in no way representative of anyone except people who are online (no surveying the Amish) and who may have a point of view to promote.
Photo courtesy of Billy Brown, via Wikimedia Commons/RNS
4chan, the anonymous online forum that delights in provoking mischief, recently upended Time magazine’s fourth annual “word banishment” online poll by encouraging people to hate on the word “feminist.”
The magazine editor later apologized for including the word “feminist” — but not for employing a survey method that’s a gateway to troll heaven.
The old-fashioned randomly dialed phone survey is biting the dust. Why? For one, consider whether you even use your smartphone for phone calls anymore. And if you’re under 30, start by Googling “landline.”
So major research firms are moving to elaborately devised panels of people drawn randomly to represent American diversity who are willing to reply online or by mail to surveys. Pew Research devised an American Trends Panel, carefully assessed so everyone isn’t the same age or inclination.
There’s a hitch, however. You can’t track change over time in surveys if the researchers changed methodology, too.
Pew Research recently released a panel-based survey on online and offline religion that found 46 percent of U.S. adults say they saw someone sharing “something about their faith” on the Internet in the last week.
Is that a greater number than five years ago? We can’t tell. Earlier surveys about religion and online behavior were phone surveys.
Cathy Lynn Grossman is a senior national correspondent for Religion News Service. Via RNS.
Got something to say about what you're reading? We value your feedback!