New series: Arbitrary yet Reasonable
I am starting a series of essays called Arbitrary yet Reasonable. It is about the many defensible decisions behind every scientific finding — which questionnaire, which model, which studies to pool — and what happens when you take all of them seriously at once instead of quietly picking one.
The first essay starts with a classic many analyst project, where 29 research teams were given one dataset, one question, and produced very different answers. From there the series walks down the research pipeline: measurement, trials, meta-analysis, and the methods for running all reasonable analyses at once — multiverse analyses, specification curves, good ol’ sensitivity analyses — including the criticisms of those methods, which I take seriously.
Alongside the essays there is a living, annotated collection of the multiverse literature, organised by where in the research pipeline the forking happens. I intend to keep it current as the field moves, and suggestions are genuinely welcome.
This site has no comment section as I will be cross-posting the essays on Substack, so if you want to discuss, argue, or tell me the turtle is standing on something after all, that is the best place to do it.