Arbitrary yet Reasonable
A series on the decisions that science rests on
Researcher A would do X, researcher B would do Y. Both are smart and capable, and ideally both come to the same conclusion — but what if they don’t? Creating science means making decisions on decisions on decisions, and many of them are arbitrary yet reasonable.
This series is about those decisions: how my thinking about them was shaped, the methods for investigating all reasonable paths at once — multiverse analyses, specification curves, good ol’ sensitivity analyses — and the criticisms of those methods, which I take seriously.
The essays
Essays appear here as they are published. The collection below is already open.
The collection
Alongside the essays there is a living, annotated collection of the literature — updated as the field moves, with notes on which analytic paths each paper varied and which decisions turned out to matter.
It is organised by where in the research pipeline the forking happens:
Layer 0 — Rationale and critiques. Should we multiverse at all? The case for running every reasonable analysis, and the strongest arguments against — that there is only one correct analysis, that multiverses manufacture doubt, that garbage specifications make garbage curves.
Layer 1 — Psychometrics. How we measure what we measure. The forking starts before any data are analysed: which instrument, which scoring theory, which measurement model — and whether that choice changes the method or the question itself.
Layer 2 — Primary studies. How we gather evidence. One trial, one dataset, and the many defensible ways to analyse it: covariates, exclusions, models, and what to do about the people who vanish halfway through.
Layer 3 — Meta-analysis. How we combine knowledge. Which studies get in, which effect size, which model, which of the many corrections for publication bias — the layer where my own multiverse work lives.
Layer 4 — Meta-meta. How we combine the combinations. Umbrella reviews, meta-meta-analyses, and what happens when the forking compounds across an entire literature.
The collection is certainly incomplete. Suggestions are genuinely welcome.