On contemporaneous controls, unlikely outcomes, boxes and replacing the ′Student′: Good statistical practice in pharmacology, problem 3

This paper is intended to assist pharmacologists to make the most of statistical analysis and avoid common errors. A scenario, in which an experimenter performed an experiment in two separate stages, combined the control groups for analysis and found some surprising results, is presented. The consequences of combined controls are discussed, appropriate display and analysis of the data are described, and an analysis of the likelihood of erroneous conclusions is made. Comparisons between data from separately conducted experimental series are hazardous when there is any possibility that the properties of the experimental units have changed between the series. Experiments that have been performed independently should be analyzed independently. Unlikely or surprising results should be treated with caution and a high standard of evidence should be required, and verification by repeated experiments should be performed and reported. Box and whisker plots contain more information than plots more commonly used to display for qualitative variables and should be used where the sample size is large enough (say, n >= 5). In most biomedical experiments the observations are not random samples from large populations as assumed by conventional parametric analyses such as Student's t-test, and so permutation tests, which do not lose their validity when a sampled population is non-normal or when the data are not random samples, should frequently be used instead of Student's t-tests.