Tiny Steps, Big Problems
·1 min
Statistical tests based on permutations (also called randomisations or resampling) are popular in many fields. In behavioural ecology and animal social network analysis permutations are used to generate reference distributions for a variety of statistical analyses. In particular, Markov chain-like algorithms are often used where samples are highly constrained (e.g. interaction matrices). In a recent tiny preprint we highlight that these permutation algorithms should be used in conjunction with MCMC diagnostics such as multiple chains, R-hat statistics, and effective sample sizes. In a realistic use case we found that nearly half of tests resulted in a false positive.