P.Mean: How much work does that second reviewer have to do in a meta-analysis (created 2011-06-20). News: Sign up for "The Monthly Mean," the newsletter that dares to call itself average, www.pmean.com/news.

Someone asked about the process of using a second reviewer in a meta-analysis to abstract data from studies. The rationale for a second reviewer, of course, is to establish that there is no serious subjectivity involved with the recording of information from individual studies. By showing that two independent reviewers produced roughly comparable data set, you have established objectivity in the data abstraction step. The question arises, though, do you have to use the second reviewer on all studies, or can you just do this for a certain percentage of the studies. If so, is there a certain percentage that is generally accepted?

As far as I know, there are no "official" rules on this. If you do a subset, there is a risk that the statistics that come out may make you wish that you had done everything with two reviewers. Don't do a subset unless you are pretty sure that the agreement will be high enough to justify not doing 100% of the data abstraction with two reviewers.

If you do a certain percentage, there are some numbers (10%, 20%) that appear often enough in the literature that you can cite them as examples. For example, "Following the process in [reference], we had a second reviewer abstract the same data from 20% of the studies. I call this the lemming school of research.

If you wanted to do this a bit more rigorously, you would justify the sample size by showing that the statistics produced by your comparison of the two raters (typically a correlation of some type) have sufficiently narrow confidence intervals.

This work is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2011-01-01. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Systematic Overviews.