Judging by the Source
“Mom, who made the first people?” he asks me sweetly. It is already nine o’clock and well past his bedtime.
Blah, blah, blah, I stumble a bit then build up to “people came from monkeys.”
“Nah,” he says and gives me a sly smile as if I’m trying to trick him. My “really, they do!” doesn’t seem to convince him.
A few days later he tells me a friend said that god made the first people, but my son informed him that he didn’t. “God made Challah,” he said. And another friend, who “knows a lot about god” has informed them both that god used to be a dinosaur. “A velociraptor!” I shared these observations with my husband, who tried to convince him. “People really did come from monkeys,” he said, but by then our son was developing complex relationships between god and Santa and who discovered the North Pole first, and, well, it seemed beside the point.
It may seem simple when interacting with a four year old to determine which assertions are true and which aren’t, but in our adult lives our ability to evaluate information directly is more limited. We often judge information by its source. We do this in medicine when evaluating scientific evidence. The articles published in medical journals, for example, can have a variety of biases. Medical school courses that teach students how to interpret the evidence and spot these biases have been added only in the recent past. But even with these sessions, the statistical methods that determine if a difference between two groups is (likely) a true difference or present just by chance outstrip the mathematical understanding of nearly all physicians. Few medical journals now publish studies that have not been reviewed by a statistician, and most researchers employ biostatisticians to aid them in data analysis and calculations. The doctor reading the article in the journal is not able to spot these biases and errors. Instead, we trust the expert reviewers, statisticians and editors of the journal where we read the article. If we are knowledgeable enough to know, we may trust the author’s reputation, or perhaps the reputation of the institution with which she is affiliated. And if we’re old enough to have seen multitudes of recommendations and correlations be overturned, we trust little.
This type of trust might seem outdated in the modern era, where we are inundated with all sorts of information and where we have the goal of shared decision-making. But in fact, it may be more important than ever. The information online, and indeed, often in the medical literature, is so overwhelming and contradictory, that even experts may disagree. And that applies to the only, by most estimates, 20% of the things we do in medicine that are actually supported by any type of scientific evidence. Medical advice is no different from other information we come across: we judge its quality by its source.
A study reported in the New England Journal of Medicine found that doctors were able to discriminate between more rigorous and less rigorous scientific studies. However, when industry sponsorship was disclosed, they were less likely to trust the results of the study, regardless of equal scientific rigor. The accompanying editorial urged physicians to “Believe the Data,” but plenty of examples of drug company misconduct of trials and scientists simply making up data would suggest that physicians, like all people, do well to evaluate what potential conflicts of interest the source of the data may harbor.
Let’s take a single recent example. Several years ago treatment with a drug showed reduced mortality in patients with severe sepsis. The drug was expensive and not without side effects – the most feared one being an increased risk of bleeding. The drug was slow to be taken up by the physician community due to further studies showing its risk profile. The pharmaceutical company, Eli Lilly, sponsored the Surviving Sepsis campaign and associated bundles of guidelines to promote increased use of the drug. They even suggested that ICU doctors were not using their drug because they were rationing care in order to pressure physicians to use their medication. In the end, the controversy led to a large randomized, controlled trial, which showed no mortality benefit. The drug was withdrawn from the market. This one anecdote is hardly an anomaly in the complex web of interests of the business of medicine.
We ought to be persuadable. We ought to be able to evaluate the facts and draw reasonable conclusions. But understanding all the ways in which the facts are not immutable, all the ways in which chance, and more sinister forces, can play a role in influencing results of scientific studies is important. Our skepticism of the latest and purportedly greatest is healthy, even if it takes us longer to sort out where the first people came from in the end.