A family of osteoporosis medications called bisphosphonates (which include Fosamax, Actonel, and others) is known to cause irritation of the esophagus. These medications have been prescribed millions of times. There are a few reports of patients developing cancer of the esophagus while taking these medicines. Obviously, many more people who develop esophageal cancer have never taken bisphosphonates. So how can we tell if bisphosphonates increase the risk of esophageal cancer?

A fascinating article in yesterday’s Wall Street Journal attempts to answer that question and highlights how the wrong kind of study can mislead us.

Let’s first make sure we understand the two kinds of studies scientists can do to try to answer the question. An observational study would involve following lots of people and comparing those who happen to be taking bisphosphonates to those who are not. No intervention is made to any patient. The fraction of people who happen to be taking bisphosphonates and develop esophageal cancer is calculated and compared to the fraction of people who develop esophageal cancer without taking bisphosphonates.

The other kind of study is a randomized study. That would involve enrolling lots of subjects and (with their permission) flipping a coin for each person. Half the subjects would receive a bisphosphonate and half would receive a placebo. The subjects would be kept ignorant of which pill they are taking. Then the subjects would be followed for many years and the number of esophageal cancer cases in each group would be counted.

Randomized studies are complex and expensive. They’re also fairly reliable and as close to definitive as anything in medicine. Observational trials are prone to countless biases that lead to spurious results. For example, smoking is a significant risk factor for esophageal cancer. If the people in an observational study taking bisphosphonates by coincidence included a higher fraction of smokers than those not taking bisphosphonates, that would lead to a misleading result that bisphosphonates increase the likelihood of cancer. Some of these biases can be accounted for with statistical techniques, but there is no way to identify or account for all of them. Randomization is the only way to try to guarantee that there is no systematic difference between the two groups except for the medicine that they are taking.

The fascinating development in the Wall Street Journal article is that two British teams attempted to answer the question about bisphosphonates by doing two different observational studies. Neither team knew about the work of the other. The studies reached conflicting results. One found a link between bisphosphonates and esophageal cancer, and the other did not. Even more disheartening is that they used the same large database of British patients to arrive at their conclusions.

The article even cites the supreme example of the folly of observational studies – estrogen. For decades, based on observational studies doctors prescribed estrogen to post-menopausal women, convinced that this prevented strokes and heart attacks. A randomized trial definitively showed that there was no benefit, and potentially some increase in risk.

Despite the notorious unreliability of observational studies, more are being done all the time. The article is frank about the reason for this – they’re much less expensive than rigorous randomized trials. But it would be even cheaper to read tea leaves, check your Magic 8-Ball, or ask my chatty neighbor. We should either invest the resources to actually learn the answer to our question, or admit our ignorance and spare ourselves and our patients the confusion that observational trials inevitably cause.

Learn more:

Analytical Trend Troubles Scientists (Wall Street Journal)