P.Mean: Statisticians are not gatekeepers (created 2008-11-04).

This page is moving to a new website.

A discussion of the proper role of statisticians when presented with questionable data is raging in the MedStats discussion group. I added some comments recently about the dangerous tendency for us statisticians to view our roles as "gatekeepers". Here's the gist of my comments.

There are many statisticians who view themselves as gatekeepers. I've heard one statistician claim that we are guardians of the scientific method, for example. That sounds great and it does help us generate pride in our discipline. But at its root, I think it is fundamentally elitist (in a bad way).

I believe that most scientists can understand the scientific method as well as we can and can judge for themselves to what extent the limitations of a research design compromises the validity of a conclusion.

I used to be a gatekeeper, but it was a dispute about the use of a Bonferroni correction about a decade ago that opened my eyes. As a younger statistician, I was very forceful in the use of Bonferroni corrections. But when I talked to researchers, I recognized that (a) they were perfectly capable of applying a Bonferroni correction themselves to a set of uncorrected p-values published by someone else and (b) they understood implicitly that a large number of outcome variables generally produced a result that was more exploratory than confirmatory.

So now, I talk to my clients about the pros and cons of various choices, but I don't refuse to work with them if they chose an approach that would not be my preference, as long as they are sincere about their preference and as long as they understand the limitations of their choices.

I still will draw a line in the sand from time to time, but I generally avoid this. My criteria for what I'll do is not "is this the correct analysis" but rather "is this a credible analysis". There's no sin in publishing a data analysis from a weak research design. The sin is in pretending that the data analysis says more than it is capable of saying.

Of course, I would always object to an analysis that is intentionally deceptive, but I can't say that I have ever worked with a researcher who desired to deceive anyone. I've worked mostly in health care, and everyone there is motivated to save lives. These researchers know that you don't save lives by producing deceptive data analyses.

This work is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2010-04-01. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Human side of statistics.