From precision medicine to personalized job training, customizing interventions for individuals is often assumed to produce better outcomes than a one-size-fits-all approach. But personalization also comes with costs: it can be more expensive, harder to implement reliably, and can require more resources to design.

Now, a new statistical test developed by Stanford researchers gives decision-makers a rigorous way to quantify whether personalizing an intervention is actually worth it. The test, called the K-fold personalization test, or KPT, was developed by Zhaoqi Li and Emma Brunskill at Stanford’s Computer Science Department. The research was published July 9 in Science.

“We’re living in the age of personalization,” said Brunskill, who is an associate professor of computer science in the School of Engineering and co-author of the paper. Brunskill and Li highlighted the breadth of applications for the newly developed test, from biostatistics to economics to machine learning. “All of these circle the same fundamental question: when does tailoring interventions to individuals improve outcomes?” said Li, who is a postdoctoral scholar at Stanford and co-author of the paper. “Our K-fold personalization test is among the first to bring these threads together into a single, rigorous test for the benefit of personalization.”

The trade-offs of personalization

Brunskill focuses on creating AI systems that can learn from a few samples to make robust decisions. As she explained, the new test helps users understand whether the benefits of personalizing a treatment are enough to justify the costs of its implementation. Given data, the test produces an estimate of the expected benefits of tailoring an intervention, along with a range for the expected benefits, called a confidence interval.

Quantifying these trade-offs is important, she noted, because as personalization adds complexity to policies, there is “more potential for that implementation fidelity to go badly.” In other words, customized plans are often harder to execute with precision than one-size-fits-all approaches.

A key insight behind the paper is that heterogeneous treatment effects – differences in how much an intervention helps different subgroups – are necessary but not sufficient to justify personalization. Even if some groups benefit more than other groups from a single, one-size-fits-all intervention, everyone still may be better off receiving that same intervention.

Brunskill used the example of job training for different age groups, where younger groups might benefit more from training than older groups. “You could have differential effects, but it might still be that everyone benefits from job training. Maybe the 18- to 20-year-olds get a lot more of a gain than the 50- to 60-year-olds, but everyone’s wages have increased,” she said.

Importantly, personalization mainly pays off when a certain subgroup is actively hurt or unserved by a one-size-fits-all intervention. “Maybe there’s an opportunity cost for 18- to 20-year-olds, and so their wages actually decrease with job training, because they’re out of the labor market,” Brunskill said. In this case, tailoring the intervention to increase the benefits for younger groups may be worth the added complexity of its implementation.

Testing interventions

The researchers developed the KPT to have controlled type 1 error, meaning it is conservative about suggesting personalization benefits that aren’t there. As Li explained, this control is important to avoid the real costs of false positives.

He discussed an example that looked at personalization of depression treatments where traditional methods for assessing the advantages of personalization falsely declared a significant benefit from personalization 7 percent of the time. “Health workers might invest substantial effort and resources into delivering tailored interventions to individuals when there is truly no personalization benefit. That would be a significant waste of effort and resources that a stable, rigorous test can help avoid,” he said.

Under some assumptions, the KPT also gives the narrowest possible confidence intervals it can, providing users a clearer picture of the range of possible benefits. Compared with existing methods, the researchers found that the KPT produced confidence intervals that were similar or narrower.

The test also highlighted instances in which personalization had minimal gains. When the researchers used the KPT to estimate the benefits of personalizing behavioral science interventions on the completion of an online course, the estimated personalization effect overlapped with zero potential benefit.

The researchers plan to release the software package for free for practitioners in fields such as social science, medicine, and education to apply the KPT to their own data. They are excited to see how their tool can translate data into real-world informed decisions that improve interventions and resource allocation. For all the work that went into developing the KPT, the team hopes adoption of it is straightforward and that it can become a go-to tool for researchers across disciplines.

“Heterogeneous treatment effect estimation is now a very common part of data analysis when estimating the impact of a potential intervention. In many settings, a key reason for this is to understand if we should provide different interventions to different groups. I’d love it if KPT could help stakeholders quantify how much benefit there would be if they were to personalize,” Brunskill said.