P.Mean: Distrust of a Bayesian meta-analysis (created 2008-07-01).

This page is moving to a new website.

A regular correspondent on the evidence based health email discussion group (BA) raised some questions about the use of a Bayesian hierarchical model in a meta-analysis. He was worried about whether this approach would be appropriate for this type of data.

There is indeed a tendency to distrust complex methods. There is perhaps a belief that the researchers tried the simple approach first, did not like what they saw, and tried some increasingly exotic approaches until they found one that produced the results they wanted. I'm sure that's not what happened here, but that is the reason I always try to use the simplest model possible (but not so simple as to do violence to the data).

My comments have to be very general, as I do not have access to this particular article.

My first rule in critical appraisal is to focus on how the data was collected and not on how it was analyzed. How data is analyzed is indeed important and they wouldn't pay me such a huge salary here if data analysis were trivial. But data collection is far more important. After all, if you collect the wrong data, it doesn't matter how fancy the analysis. Furthermore, nine times out of ten, if there is a flaw in a paper it is a flawed data collection rather than a flawed data analysis.

Furthermore, we have to trust the peer review process to some extent. There are many technical aspects to how a research study is conducted, not just the statistics, and we have to believe that the journal has selected peer-reviewers with appropriate technical expertise.

For what it's worth, Bayesian methods are becoming bigger, and meta-analysis is one area in particular where we are likely to see more of this approach. The Bayesian model generalizes very well to more complex settings and the classical approach often forces you to make simplistic assumptions.

The most recent Bayesian approaches rely on simulation methods with exotic names like Metropolis-Hastings algorithm, or Gibbs sampler, or Markov Chain Monte Carlo. For those who are curious about the technical details, I do have a very simple illustration of the Metropolis-Hastings algorithm. Most of you probably don't want to know this. I close my eyes when they show pictures of actual surgeries, so you are entitled to close your eyes when random distributions are simulated on the computer.

There is an inherent distrust of simulation models. Look at all the controversy about simulation models for global warming. I would argue that the Bayesian simulation methods have been vetted enough by people much smarter than me to have excellent credibility. You don't want amateurs in charge of this type of model, though, so be sure that one of the authors or someone mentioned in the acknowledgments has a background in Statistics.

The bottom line: if there are no problems with how the studies were selected, then I would not quibble about the Bayesian analysis.

This work is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2010-04-01. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Bayesian statistics.