September 27, 2006
Social science researcher proposes overhaul of survey methodology
By Lisa Trei
Armed with a new $2 million grant from the National Science Foundation's Major Research Instrumentation (MRI) Program, Jon Krosnick plans to introduce sweeping changes to traditional survey methodology. If successful, Krosnick hopes his project will enable representative sample surveys to obtain accurate, high response rates with quick turnaround times at reasonable costs.
By marrying Internet survey methodology with old-fashioned, face-to-face interviewing, Krosnick's two-year project will explore whether it is possible to create a computer network equivalent of an "astronomer's telescope," providing shared collection and analysis of social-science data. If it works, the result will allow groups such as academics, government agencies and businesses to share the expense of sample recruitment for a range of projects.
"The logic only hits you when you think of it," said Krosnick, a professor of communication and of political science who holds the Frederic O. Glover Professorship in Humanities and Sciences. "People are so entrenched in their habits, they probably haven't thought of it."
Historically, Krosnick explained, scientists have used MRI competition grants to purchase books or large pieces of hardware, such as telescopes. "This is an important grant because social scientists say they need something like a telescope too," he said. "What is a telescope? It is a measuring device that is made available on a shared basis to lots of different researchers. That is exactly what this [project] is."
According to Krosnick, during the last three decades surveys have become increasingly central to decision-making in many aspects of life. Government agencies, companies and academic scholars spend billions of dollars to obtain survey data, he said. "Everybody has come to realize that in order to stay in touch with the realities of the marketplace, in order for government to be responsive to its citizens and in order for academics to understand what drives people's behavior, surveys are an incredibly efficient way of getting at data."
Despite this growing demand, Krosnick said, survey response rates have declined during the last decade as the Internet and cell phones have made people more accessible but less available. "In the 1970s, you could get a telephone-survey response rate of 70 percent," Krosnick said. "Now, if you work really hard, you might get 40 percent." Surveys on the front pages of major newspapers have response rates of less than 10 percent, he added.
It is still possible to conduct high-quality surveysface-to-face interviews yield 80 percent response ratesbut such methods cost $1,000 per subject Krosnick said. Telephone interviews cost $2.50 to $6 a minute, he said, but respondents, even if they are available, usually won't talk on the phone for more than 20 minutes. And while research shows that people answer questions by computer more accurately than by telephone, 90 percent of Internet surveys have self-selected respondents, Krosnick saidwhat is termed "haphazard sampling" of volunteers.
"The question is, how do we get out of this mess?" he said.
Krosnick's solution is to have statisticians at RTI International, a leading survey research organization, draw up a representative sample of 1,000 American households based on U.S. Postal Service mailing lists. RTI staff then will visit the households, randomly select an adult member, conduct a brief face-to-face interview and offer a free laptop and high-speed connection in exchange for respondents answering a 30-minute, secure Internet survey once a month. Evaluations of the network will gauge the accuracy and frequency of response rates and the attrition of participants, he said. Matthew DeBell, a staff researcher at Stanford's Institute for Research in the Social Sciences, will supervise the RTI statisticians involved with the project. Households will be recruited in March and data will be collected starting in September 2007.
2008 national election and beyond
Initially, the computer network will be used to study the dynamics of the 2008 presidential primaries and general election campaign, Krosnick said. Scholars will submit proposals to the American National Election Studies, which has conducted surveys of the U.S. electorate since 1948, to gauge respondents' views of the country, political candidates and election process. Afterward, Krosnick said, minutes on the monthly survey could be sold to pollsters interested in researching anything from health and education to consumer behavior.
If the proposed network works, Krosnick said, it would allow agencies to share the costs of sample recruitment across many projects, thus yielding higher response rates and lower costs. The Office of Management and Budget, which approves all federal surveys, requires high response rates to obtain government support.
"Everybody has been pulling their hair out with unhappiness over the mounting problems with survey methodology," Krosnick said. "This is the only innovation that shifts gears. What's nice about this project is that, for the first time, it says to the world that Stanford is the center of the most exciting innovative steps being taken in survey methodology."