Statistical Evidence. Chapter 4. What do the other witnesses say? Corroborating Evidence.

4.0 Introduction

In a criminal trial, the prosecutor will sometimes try to demonstrate that the defendant had:

All three elements are not really necessary for a conviction--many people are convicted without the need to show a motive for example. But when the prosecution can identify a motive, that makes their case that much more convincing.

This analogy also holds for research studies. Some studies are so well done that their evidence alone would be enough to convince you. Other studies, however, provide only weak evidence. But when this evidence is combined with other information, the evidence can become quite strong.

Sir Austin Bradford Hill outlined a series of tests that you could use to evaluate whether an association between an environmental factor and disease was credible (Hill 1965). These criteria are not perfect. A strong criticism of Hill's criteria appear in a classic textbook on Epidemiology by Kenneth Rothman and Sander Greenland (Rothmann 1998). Rothman and Greenland point out that none of Hill's criteria (with the exception of temporality) are necessary or sufficient for establishing cauastion.

While their criticisms are important to remember, I believe they missed the point of Hill's original article. No one criterion by itself will establish the credibility of a research study if it is present and no one criterion will destroy the credibility of a study if it is absent. You should look at the aggregate impact of these factors. When most of them are present, they add to the credibility of a study. When they are absent, they weaken the credibility of the study. As Sir Austin Bradford Hill himself notes:

"All scientific work is incomplete - whether it be observational or experimental. All scientific work is liable to be upset or modified by advancing knowledge. This does not confer upon a freedom to ignore the knowledge we already have, or to postpone the action that it appears to demand at a given time. Who knows, asked Robert Browning, but that the world may end tonight? True, but on available evidence, most of us make ready to commute on the 8:30 next day."

Case Study: A drug treatment that only works in black patients

There has been a lot of published research that shows that heart disease is different and more deadly among black patients. Some possible explanations of these differences involve the renin-angiotensin system in bioavailability of nitric oxide. In a study that seemed to show no overall differences in efficacy for a drug treatment, hydralazine plus isosorbide dinitrate, for treating congestive heart failure, there was nevertheless the suggestion that this treatment might be effective when analysis was restricted to just the black patients in the study. This study, however, was not designed to look for race-specific effects, so the results had to be treated as preliminary. The authors of one review state  that "prospective trials involving large numbers of black patients are needed to further clarify their response to therapy" (Carson 1999). With this justification, a new randomized trial, recruiting just black patients, was begun. This study did indeed show that the two drugs were effective among these black patients (Taylor 2004), and became one of the first examples of a therapy recommended solely for a specific racial subgroup.

The concept of using race or ethnicity in medical decisions is controversial, because of the potential for misuse and abuse of this information (Bhopal 1997). There is also debate about whether there is enough genetic variations among different racial and ethnic groups to justify treating them as distinct group. The authors of the second study skirt this issue by using the phrase "patients who self-identify as black".

The important lesson, though, is that no study should be examined in isolation. You should always be looking for corroborating evidence. The subgroup finding in Carson 1999 was indeed a weak form of evidence, but it was supported by several mechanistic explanations described above. When these results were replicated in an independent study, the evidence in favor of this controversial treatment became overwhelmingly persuasive.

What do the other witnesses say? What to look for.

Additional details, both within the research study itself and outside the research study can provide support for an otherwise weak form of evidence.

Is there a strong association? A treatment that has a large impact is unlikely to become undone by small flaws in the research.

Is there a dose-response pattern? A treatment that shows stronger effects when given in stronger doses adds credibility to a study because it reduces the credibility of certain biasing factors.

Is the association consistent? A result that is replicated across diverse populations using diverse research designs adds credibility because it is unlikely that a particular flaw in the research could affect all these studies in the same way.

Is the association specific? A treatment that cures "everything" lacks specificity. You should mistrust such a treatment because it is likely to be caused by a global difference in the health of the treated and untreated patients. In contrast, a treatment that cures one particular condition, but not others would rule out such a global difference.

Is the association biologically plausible? A treatment that has no sound biological basis has to pass a higher threshold of evidence than a treatment that has a plausible biological mechanism.

Is there a conflict of interest? Research that is untainted by commercial temptations is more credible because the researchers have no financial incentive to skew the research results.

Is there any evidence of fraud? Research that is carefully reviewed reduces the chances of deliberate falsification of the data.

[Remainder of this material deleted out of respect for the publisher's copyright.]

Creative Commons License This work is licensed under a Creative Commons Attribution 3.0 United States License. It was written by Steve Simon on 2005-06-10, edited by Steve Simon, and was last modified on 2017-06-15. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page. Category: Statistical evidence