StATS: Reviewing a paper on qualitative data analysis (March 11, 2007). Category: Qualitative data analysis

I was asked by BMJ to review a paper that involved a qualitative data analysis. These reviews are confidential, so I don't want to describe the paper in any detail. It is worthwhile, however, to note some of the standards that others have suggested for assessing the quality of a qualitative data analysis.

This is important for a peer-reviewer like myself. I need to be able to assess whether the authors have produced a result that is sufficiently rigorous to merit publication.

Perhaps the best paper in my files on this topic is

The authors address a debate in the research community: should qualitative research be judged by the same standards as quantitative research. In particular, do concepts like validity, generalizability, and reliability apply in the same ways?

Those who would answer "no" to this question, the antirealists,

argue that qualitative research represents a distinctive paradigm and as such it cannot and should not be judged by conventional measures of validity, generalisability, and reliability. At its core, this position rejects naive realism---a belief that there is a single, unequivocal social reality or truth which is entirely independent of the researcher and of the research process; instead there are multiple perspectives of the world that are created and constructed in the research process.

The authors do not use the term, but the perspective of antirealists reminds me of post-modern philosophy. There are numerous post-modern critiques of evidence-based medicine, and I would like to summarize some of these articles in a separate web page.

On the opposite side of the fence are the relativists, those who would offer a conditional "yes" to the above question. According to the authors, relativists believe that

assessment criteria are feasible but that distinctive ones are required to evaluate qualitative research have put forward a range of different assessment schemes. In part, this is because the choice and relative importance of different criteria of quality depend on the topic and the purpose of the research.

The criteria used by relativists to assess the quality of a qualitative analysis include the following:

 I personally do not find this list very helpful.

The authors then summarize a perspective of a third group, the subtle realists, who hold a position somewhere between antirealists and the relativists. Like the antirealists, this group does recognize that

there are multiple perspectives of the world that are created and constructed in the research process,

but does argue that

there is an underlying reality which can be studied

and that while research may not be able attain the "truth", it is capable of representing the underlying reality.

The difference between attaining truth and representing an underlying relative is a subtle difference, and it is unclear to me whether this in an important difference. But the criteria that the authors propose for this group are promising.

The authors do point out that all of these criteria have limitations, but I believe it still represents a valuable and interesting list of perspectives to consider as you are reviewing a qualitative research study.

Outside resources:

The last source is particularly relevant to me.

Creative Commons License This work is licensed under a Creative Commons Attribution 3.0 United States License. It was written by Steve Simon.

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at