Long COVID is a phenomenon where serious health effects linger well after a patient tests negative for coronavirus infection. Patients describe persistent “brain fog,” fatigue, dizziness, rapid heart rate, shortness of breath, and chest pain, among symptoms that affect nearly every organ system in the body and that last a year or more after the infection has passed.

In a new study, data scientists at Stanford University combed a massive dataset of 14.4 billion health insurance records from more than 244 million Americans and found that conventional methods may substantially overestimate the range and duration of health effects attributable to COVID-19.

“Our new study design found 98 percent fewer long-COVID-attributable outcomes between four to 12 months after infection compared with conventional approaches applied in the same dataset,” said co-senior author Jure Leskovec, a professor of computer science. “This doesn’t mean long COVID is 98 percent less common. It means that how we design these studies fundamentally changes which health effects we can reliably attribute to COVID.” The results are published in the journal Proceedings of the National Academy of Sciences (PNAS).

Improving population-level studies

Beyond its implications for understanding long COVID itself, this research carries an important message about how researchers can better design future population-level health studies, Leskovec said.

Improvements suggested by this work could help reduce biases that lead to skewed or unreliable results. Huge quantities of data are valuable to the extent they can reveal subtle patterns and help rule out implausible associations, but scale alone cannot overcome bias, the authors say.

“Even with 244 million patients, you can get highly precise answers to the wrong question,” said Leskovec. “Study design matters as much as data size.”

Scientists need to be as rigorous, careful, and thoughtful as physicians, argue the authors. They must test for positive associations, Leskovec explained, but also run control checks and explore negative associations as a stress test on their assumptions and approaches.

“This study used a synthetic-control design that is popular in economics and in the social sciences to align patient histories for cases and controls,” said co-senior author Stefan Wager, an associate professor of operations, information, and technology at Stanford Graduate School of Business. “It was exciting to see its promise applied to a large-scale medical study. I’m hopeful that there are many other applications where synthetic-control methods can yield valuable insights when applied to electronic health records or claims data.”

“You can take this messy, noisy data and, with the right study design, get real signal out of it,” said first author Hamed Nilforoshan, a doctoral student in computer science working in Leskovec’s lab. “Data at this scale was basically out of reach for clinical research before, because people didn’t trust it. We show it can work.”

Bias revealed

Nilforoshan first set out to reproduce the results of other high-profile long-COVID studies that suggested the disease had an unusually broad range of symptoms, some persisting for extended periods. He was able to match their findings exactly if he used their approach. But then he started to dig deeper. He looked to see whether long COVID appeared to cause impossible outcomes – lightning strikes, dog bites, and gunshot wounds – and found that it did.

“Something wasn’t adding up,” Nilforoshan said.

The problem was a phenomenon known as confounding bias – systematic differences between the compared populations that also affect the outcome. Earlier studies compared people who tested positive for COVID against people who were never diagnosed, which includes people who were not tested. The flaw in that design is that people who choose to get tested differ systematically from those who do not, which was confirmed in this study. They may have underlying conditions, non-COVID infection, or just seek medical care more frequently, creating misleading associations between COVID and subsequent health problems if that is ignored.

Leskovec clarifies the subtlety: “If you compare people who tested positive for COVID with people who were never tested, you risk attributing differences in their underlying health and healthcare-seeking behavior to the virus itself.”

Instead, the researchers selected only patients getting their first PCR test for COVID and looked at the long-term outcomes for those who tested positive against those who tested negative. This helped them isolate COVID’s real symptoms and to measure their duration more accurately.

Then they used negative controls to stress-test their hypotheses by returning to Nilforoshan’s aforementioned unlikely outcomes – lightning strikes, dog bites, gunshots, and so forth.

“A reliable method should not identify those outcomes as the effect of long COVID,” said Leskovec. As expected, those symptoms were not flagged. “These negative controls give us greater confidence that we’re identifying genuine COVID-attributable effects,” Leskovec explained. The study listed myopathy, non-TB pneumonia, respiratory issues, and pneumothorax among top COVID-attributable effects it verified.

“These findings indicate that long COVID may be more similar to other extended post-viral syndromes than many have thought,” said author Steven Goodman, professor of epidemiology and population health in the Stanford School of Medicine. “From a scientific perspective, this was a true team effort. Harmonizing the research cultures and methods of computer science, statistics, and epidemiology was essential to providing this important evidence on the nature of long COVID, yielding a product better than any of us could have created alone.”

Real-world advice

For doctors and patients, the study changes little. Long COVID is real. It can last months and affect multiple organ systems. The study suggests that the additional health risks attributable to COVID diminish more quickly at the population level than earlier estimates indicated, although some patients may continue to experience symptoms for a year or longer.

The researchers caution that insurance records cannot capture every patient experience, particularly symptoms such as fatigue and cognitive difficulties that may go undiagnosed or unrecorded.

“These results should complement physicians’ clinical assessments, not replace them,” Leskovec says. “On the other hand, for data scientists, earlier long COVID studies might need to be revisited with better controls. Our approach could improve those and future population-health studies.”