P.Mean: Can I focus on a subset of conditions in my experiment? (created 2013-09-10). News: Sign up for "The Monthly Mean," the newsletter that dares to call itself average, www.pmean.com/news.

I got an email from a colleague who was running a 2 by 2 by 6 factorial design. Without giving up too much detail, let's say that the first factor was protein (Y or N), the second was oxygen level (normal or high) and the third was media type (A, B, C, D, E, and F). This person wanted to analyze a subset of factors. He wanted to look at the combinations protein=Y/oxygen=normal, protein=Y/oxygen=high, and protein=N/oxygen=high, but did not seem interested in looking at protein=N/oxygen=normal combination. Also he wanted to compare only four of the media types (B, C, D, and E). Was his approach valid?

Well, I always worry about the word "valid" because it means different things to different people. What I typically look at is whether an analysis strategy will be considered acceptable by a peer reviewer. I know how peer reviewers think, having done it a few times myself.

The problem with the approach you suggested is that you will come across as inconsistent to a peer reviewer. There's nothing wrong with being inconsistent here, but you need to make a case for why you are being inconsistent. There's cute joke about how a statistician is someone who doesn't believe in America because it wasn't in Christopher Columbus's original research plan. Only a fool is perfectly consistent all the time.

You have to address the question: why did you set up an experiment with 2*2*6 different conditions, if you didn't plan to analyze some of those conditions. Perhaps Media=A was a positive control and Media=E was a negative control. You ran these to assure assay sensitivity. If there was not a sharp jump between A and E, then the experiment was compromised.

Perhaps you are skipping the Protein=N/oxygen=LO combination because that represents a negative control group since there is nothing being manipulated. this is another quality check. If the six media types are not flat and low for the untreated combination, then you have a quality control issue, such as contamination.

You need to articulate your rationale reason clearly. For example, why was it important to run assay sensitivity for each protein/oxygen combination? You want to make sure that the choices you made during the design of the experiment as well as the choices that you made during the analysis of the experiment are justified. You want to avoid the accusation that your choices are arbitrary or capricious or worse your choices were made to skew the research in a particular direction.

Your justification will, by necessity be post hoc. You didn't specify this particular analysis plan during the protocol development. I know this because anyone who had the foresight to do this would not be coming to me for advice. A pre-specified analysis plan would have saved you a lot of grief right now. You didn't do it, though, so you have to develop the justification for your analysis strategy post hoc.

Post hoc may sound bad, but remember the Christopher Columbus's discovery was also post hoc. Post hoc has its limitations and they are often severe limitations. But that does not mean that post hoc is always to be avoided.

It is very important, though that the post hoc justification cannot be based on the observed data. You can't say "These means looked different and those means did not, so that's why we only tested the first group." This is known by a couple of names: "cherry picking" or "data dredging". And it is generally frowned upon. Some reviewers will allow this if you label your study as hypothesis generating or exploratory, but you are fighting an uphill battle. You chose a fairly complex research design, and it's hard to argue that such a design was for predominantly exploratory purposes.

So you need to come up with a justification that you could have used prior to collecting the data. Something that could have been written up in the protocol if you had had the foresight to do prior to data collection. It's a post hoc marshalling of arguments that could have been made a priori. Will a peer reviewer buy this? It's hard to say.

If you decide to go down this route, you need to consider adjustments for multiple comparisons. This gets tricky, and there is no research consensus about when and how you should use multiple comparison adjustments. One approach is to apply a Bonferroni correction for the three different protein/oxygen combinations. Then apply another multiple comparisons adjustment for the six possible pairwise comparisons among the media types B, C, D, and E. This could be done using the Tukey post hoc test, but it is a bit tricky to layer Tukey on top of Bonferroni. It might be a bit cleaner to use Bonferroni twice. You are adjusting for six pairwise comparisons within three treatment combinations, so divide your alpha level by 18.

You might argue that you do not need to adjust for the three treatment combinations. This will gain you a bit of power and precision, but it is difficult to justify. Look for existing publications that use this precedent and be ready to cite them if your peer review report comes back critical of your approach.

Alternately, you could be ultra conservative and adjust as if you had included the fourth treatment combination and pairwise comparisons among all six of the media types. This would require you to divide your alpha level by 90 (4 protein/oxygen combinations times the 15 pairwise comparisons among the six media). This may seem extreme, but if your justifications of looking only at particular combinations is weak at best, then choosing an ulta conservative adjustment might be seen as compensatory.

There are lots of other possibilities here. In fact there are some fairly standard ways to approach this type of design. You could look at a main effects and interactions model and only investigate particular protein and oxygen combinations if a significant interaction warranted that level of investigation. Negative controls could be incorporated into the analysis as contrasts. You'd be looking not at the effect of media types in isolation but rather as how much they increase the signal over your negative control media A.

As with all questions like this, I hesitate to make any firm recommendations withoug understanding more about your particular experiment. Take any advice I offer with a grain of salt. Perhaps by outlining some of the possible choices, that would help you make some intelligent choices about your analysis strategies.

This page was written by Steve Simon and is licensed under the Creative Commons Attribution 3.0 United States License. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Analysis Of Variance.