June 15, 2020
Figuring out what works
Healthy people follow health trends. That behavior can actually cloud research.
Healthy people tend to follow health advice, which can complicate research on how health behaviors affect outcomes.
Bigstock
Every day it seems there’s new diet advice promising improved health and longevity. Often, it’s contradictory.
A new study says we should go vegan. But wait, maybe it’s not the best. Avoid fat. Actually, fat is good for you. Sugar is the real problem. People who drink coffee live longer. Coffee will kill you.
Health-conscious consumers follow this advice, perhaps believing that it’s valid if there’s a “study” behind it. Ironically, the fact that healthy people follow health trends is inadvertently clouding the research, according to a paper in the American Economic Review: Insights.
“There are different things that happen in the medical system where we all sort of decide that something is good for people because the people that have had that are having better outcomes,” author Emily Oster told the AEA. “But in fact, the outcomes are due to something else.”
Oster’s paper underscores the complexity of health research and the importance of reading these studies with a critical eye.
It’s not surprising that healthy people are going to have healthy lifestyles, and that it could be any number of different behaviors—regular exercise, rest, a vitamin-rich diet—that would lead to a particular outcome. But Oster wondered whether these dynamics could be self-reinforcing, with enough healthy people responding to advice to throw off the results in a meaningful way. If so, it would be difficult to correct the kinds of statistical accidents that result in false positives or negatives.
There are different things that happen in the medical system where we all sort of decide that something is good for people because the people that have had that are having better outcomes. But in fact the outcomes are due to something else.
Emily Oster
Oster looked at swings in the recommendations for taking vitamin E. In the 1990s, a number of studies pointed to large benefits from taking those supplements in preventing heart disease. But that changed in 2004, when new evidence emerged that said taking vitamin E not only didn’t help with heart disease, but taking it in excess could actually shorten someone’s life.
“There were these two very sharp changes, and that's a particularly good laboratory to look at these issues,” Oster said.
Using data on vitamin supplementation and diet, she found that purchases and consumption of vitamin E spiked in the 1990s and fell dramatically after 2004, driven mostly by wealthier and healthier individuals. The people who were responding the most to health advice had higher education and income levels, were more likely to exercise and have a healthier diet, and were less likely to smoke.
So, if there were a bunch of people who were already more inclined to be healthy and they all suddenly started taking vitamin E, that then kicks off a vicious cycle in which the spurious findings in one study are more difficult to detect in follow up research. More of the heart-healthy people were taking vitamin E, so subsequent research showed the link to appear stronger than it actually was—at least, until the advice changed.
Oster also looked at other changes in recommendations for vitamin D, sugar and fat, and the Mediterranean diet. All had similar patterns. The changes in behavior were throwing off the estimated impacts on health outcomes.
There are some key lessons for anyone reading new research, and especially for journalists who are presenting these findings to the public. They need to read health research with greater skepticism and place it in the appropriate context.
A news publication will post a story about the health benefits of coffee, only to publish another story three weeks later that says the opposite, and with little mention about how these studies stack up to the hundreds that have come before.
“There's no delving into whether some of these studies are better than others,” Oster said. “I think there's a lesson that actually these problems of bias are not just something you put in a limitations section. . . In a lot of these cases, bias is the whole thing.”
♦
"Health Recommendations and Selection in Health Behaviors" appears in the June issue of the American Economic Review: Insights.