Social science researcher to overhaul survey methodology with $2 million grant
Armed with a new $2 million grant from the National Science Foundation's Major Research Instrumentation (MRI) Program, Jon Krosnick plans to introduce sweeping changes in traditional survey methodology. If his project is successful, the findings will allow representative sample surveys to obtain accurate, high response rates with quick turnaround times at reasonable costs.
By marrying Internet survey methodology with old-fashioned, face-to-face interviewing, Krosnick's two-year project will explore whether it is possible to create a computer network equivalent of an "astronomer's telescope," a shared platform many investigators can use to collect social science data efficiently. If it works, the result will allow academics, government agencies and businesses to share the expense of sample recruitment for a range of projects.
"The appeal of this new method grows on you once you start to think about it," said Krosnick, a professor of communication and of political science who holds the Frederic O. Glover Professorship in Humanities and Sciences. "Most researchers are so entrenched in their data collection habits that they probably haven't even considered this new approach yet."
Historically, Krosnick explained, scientists have used MRI competition grants to purchase large pieces of hardware, such as telescopes and boats. "This is an important grant because social scientists need their own telescopes too," he said. "What is a telescope? It is a measuring device that is made available on a shared basis to lots of different researchers. That is exactly what our survey platform will be."
According to Krosnick, during the last three decades surveys have become increasingly central to decision-making in many aspects of life. Government agencies, companies and academic scholars spend billions of dollars to obtain survey data, he said. "Everybody has come to realize that in order to stay in touch with the realities of the marketplace, in order for government to be responsive to its citizens and in order for academics to understand what drives people's behavior, surveys are an incredibly efficient way of getting data."
Despite this growing demand, Krosnick said, survey response rates have declined during the last decade as the Internet and cell phones have made people more accessible but less available. "In the 1970s, you could get a telephone-survey response rate of 70 percent," Krosnick said. "Now, if you work really hard, you might get 40 percent." Surveys on the front pages of major newspapers have response rates of less than 10 percent, he added.
It is still possible to conduct high-quality surveys—face-to-face interviews yield 80 percent response rates—but such methods cost as much as $1,000 per subject, Krosnick said. Telephone interviews cost $2.50 to $6 a minute, he said, but respondents, even if they are available, usually won't talk on the phone for more than 20 minutes. And while research shows that people answer questions by computer more accurately than by telephone, 90 percent of Internet surveys have self-selected respondents, Krosnick said—what is termed "haphazard sampling" of volunteers.
"The question is, how do we get out of this mess?" he said.
Krosnick's solution is to have statisticians draw up a representative sample of 1,000 American households based on U.S. Postal Service mailing lists. Research staff then will visit the households, randomly select an adult member, conduct a brief face-to-face interview and offer a free laptop and high-speed connection in exchange for respondents answering a 30-minute, secure Internet survey once a month. Evaluations of the network will gauge the accuracy and frequency of response rates and the attrition of participants, he said. Matthew DeBell, a staff researcher at Stanford's Institute for Research in the Social Sciences, will supervise the fieldwork involved with the project. Households will be recruited in March and data will be collected starting in September 2007.
2008 national election and beyondInitially, the computer network will be used to study the dynamics of the 2008 presidential primaries and general election campaign, Krosnick said. Scholars will submit proposals to the American National Election Studies, which has conducted surveys of the U.S. electorate since 1948, to gauge respondents' views of the country, political candidates and election process. Afterward, Krosnick said, minutes on the monthly survey could be sold to researchers interested in studying anything from health and education to consumer behavior.
If the proposed network is successful, Krosnick said, it would allow agencies to share the costs of sample recruitment across many projects, thus yielding higher response rates and lower costs. The Office of Management and Budget at the White House, which must approve all federal surveys, requires high response rates to obtain government support.
"Everybody has been pulling their hair out with unhappiness over the mounting problems with survey methodology," Krosnick said. "This is the only innovation that shifts gears. What's nice about this project is that, for the first time, it says to the world that Stanford is the center of the most exciting innovative steps being taken in survey methodology."