Statistical analysis plan giving away some of my results, Reviewer 2

This is a ridiculous request (edit: eh, maybe not quite so ridiculous; comments suggest that my 'however' paragraph below applies in this case, but I'll leave the rest of this answer as-is). For one, even before you have run any tests you should have a plan for post hoc testing.

I would respond that it's not possible to correct this without moving statistical methodology to the results, and that the authors feel strongly about keeping this methodology to one section. I can't imagine an editor having a problem with this response, and if the reviewer feels strongly that the boundaries between sections of a paper are sacred then this speaks to their preference directly as well.

However, make sure you aren't saying things you don't need to say if it will help please the reviewer. For example, if you write something equivalent to "we do ANOVA on apple oranges and banana models" and then say "for oranges, we then did post hoc test xyz corrected for multiple comparisons", you can drop oranges from the second section, writing "if significant, we then did post hoc test xyz corrected for multiple comparisons". It doesn't matter that you ran these tests on oranges, it matters that you planned to do this post hoc testing for any significant omnibus ANOVA results, no matter which class of fruit.


Without seeing the reviewers' exact comments, it's hard to say, but I suspect that they may simply want to you be clear about what you decided to do before performing the experiment, and what you decided to do after performing the experiment. In "run-of-the-mill" science, we form a hypothesis, design an experiment to test the hypothesis, and then perform the experiment. The order in which these things happen is very important when interpreting the strength of the results. Usually, the "Methodology" section in a paper describes the design of the experiment: i.e. the stuff that you decided to do before performing the experiment. If you put stuff out of chronological order, then it can be difficult to determine whether you decided to do activity X before or after previewing the results.

I'm not suggesting that it is absolutely crucial that you organize your paper in this fashion, but I think that this is the crux of your reviewers' concerns. So, you can probably appease them by adding clarity surrounding the chronology of events: If you have hypotheses that you formed before the experiment and hypotheses that you formed "during" the experiment be clear about this; if you changed course in your experiment because of preliminary results - be clear about what the original plan was, what the amended plan is, when in the experiment the amendment occurred and why.