There are many ways to skin a cat, and turtles all the way down
A few years ago one of my favorite studies was conducted: 29 teams of smart, well educated researchers were asked to investigate the same question – are referees racist and give more red cards to darker skinned players? In an ideal world, from a human standpoint, there would be no racist referees and from a scientific standpoint, the results would be robust: all research teams would come to the same or a very similar conclusion, either yes they are racist, or, no, they are not racist. Or maybe even it’s inconclusive. But in the study they found all different answers, some said they are racist, some said they aren’t, and some said it’s hard to say. How can this be? Why are there so diverging conclusions? Each team made defensible, yet different analytic decisions.1
Ever since reading about this study, I think about these diverging results that can arise on the same question basically every day, as I believe it not only applies to racist (or not) referees. It applies to every scientific question. Researcher A would do x, researcher B would do Y. Both are smart and capable. Ideally both come to the same conclusion, but what if they don’t? Researcher C does Z? I have been involved in research projects spanning questions on how to measure invisible constructs like depression or health related quality of life; how to investigate the effectiveness of psychological interventions; how to synthesize many published findings and extract the statistical gist out of them; and meta research, investigating how research is done. All of those research areas contain questions that have so many different ways to go about them, so many arbitrary decisions can be made.2 There are many ways to skin a cat (I hope this saying exists and is appropriate, I tend to mix German and English sayings, so apologies if culturally inappropriate. I am a dog person).
You can measure depression with three hundred different questionnaires, which one should a researcher pick?3 How do you handle outliers or extreme values on those scales in your study? What do you do with those participants that drop out? Each of those can be answered in very different yet reasonable ways. Then there are thousands of statistical models you can run on basically any research question, and there are so many corrections and adjustments statisticians can make. And when you take hundreds of such studies, you can pool and average them in hundreds of ways, and many are really reasonable and smart to do. And the thing is: researchers almost never agree on the best way to do it, so there are dozens of arbitrary yet reasonable decisions in those meta-analyses, containing studies that contain dozens of different arbitrary yet reasonable decisions, using instruments that were created with dozens of arbitrary yet reasonable decisions, which were informed by meta-analytic evidence, with all those arbitrary yet reasonable decisions, which … you get the point, it’s turtles all the way down.
The “turtles all the way down” saying is something that is attributed to, as all good sayings: nobody really knows, and I retell it here without looking it up properly to add to the mystery: Either William James or Bertrand Russell or another glorified hero of reason gave a lecture on the universe, what else, and an old Lady said something like: this is all very interesting but wrong, the universe rests on a turtle. The genius asked, quite arrogantly (in my mind in a posh English accent, even though William James was American and Bertrand Russel wasn’t posh, so it needs to be more an archetype of the genius yet arrogant academic): what is the turtle standing on then, honey? The old Lady smiled: ahh, bless your heart, another turtle, honey.
I love that old hag, no undue respect for authority, very opinionated, she is the hero of this series, as I believe she is right on the eternal regress we can find in the process of creating science. It’s turtles all the way down. It’s decisions on decisions on decisions, and many of them arbitrary yet reasonable.
Most of my research is influenced by this idea of investigating all reasonable paths and figuring out if they all lead to the same conclusion. In my PhD defense I made the argument that a good hiking route system would lead to the peak of a mountain you want to climb, and if people end up on different mountains, there might be a problem with the way the routes were designed.
In this series I write about how my thinking of these many arbitrary yet reasonable decisions was shaped, will dive deep into the methods within this world, from multiverse (meta)analyses (a single possible analytical path represents a universe that is contained in a multiverse of an infinite number of other possible universes on that same research question), specification curve analyses, good ol’ sensitivity analyses.4 I will dive deep also into the criticisms of the method and how I try to continually implement those identified weak points of multiverse-style analyses in my own work. I absolutely agree with those that belittle multiverse analyses as a weak, gutless move to not commit to one proper way of doing science.5 There might exist a best way or path to answer a question, however, who is the person to decide what that is? And sometimes even consensus on what that path might be is not necessarily the best way to find it (even though I am a strong believer in cumulative science). During Ignaz Semmelweis time it was not common knowledge that doctors might be killing mothers and babies if they deliver baby’s with unwashed hands coming straight from the morgue. Consensus was that babies and mothers just die for mysterious reasons (again, writing this from memory, to add to the mystery). Ignaz Semmelweis died in an asylum as he was an outcast with his outlandish idea that washing hands could save them. His contrarian personality allowed him to challenge the consensus of his time, yet also made him quite insufferable and ultimately ineffective, he called his colleagues murderers, so that nobody wanted to listen to him. A destiny of many contrarians.
I do believe that the old hag was such a contrarian, and that reminding us that the floor is resting on turtles was a brave and beautiful act. There are many ways to build a rocket (I assume), and when they fly, they fly. No matter the method. Very easy to see if they figured something out about rockets that is true. In sciences with harder to measure failures of their scientific process than “the rocket unfortunately blew up, we made a mistake”, basically every other scientific endeavor from social sciences to medical sciences, we need robust findings, that point all on the same direction, whether something is working or not, whether someone is racist or not, whether we found out something true about reality, or not.
AI disclaimer: These posts are written by me without the use of LLMs. A previous version was polished by Claude Fable 5, but it lost everything that made my voice my own, and it gave me the ick.
Layer 0 — Rationale and critiques. The annotated literature behind this essay — the argument about whether we should multiverse at all — lives in the living collection.
Footnotes
Silberzahn, R., Uhlmann, E. L., Martin, D. P., et al. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646↩︎
Many definitions of arbitrary exist, I prefer this one, so I use it throughout, arbitrarily:
↩︎More than 280, to be precise: Santor, D. A., Gregus, M., & Welch, A. (2006). Eight decades of measurement in depression. Measurement: Interdisciplinary Research and Perspectives, 4(3), 135–155. https://doi.org/10.1207/s15366359mea0403_1. And the scales overlap surprisingly little in content: Fried, E. I. (2017). The 52 symptoms of major depression: Lack of content overlap among seven common depression scales. Journal of Affective Disorders, 208, 191–197. https://doi.org/10.1016/j.jad.2016.10.019↩︎
The founding papers of this family: Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712. https://doi.org/10.1177/1745691616658637; and for meta-analysis: Voracek, M., Kossmeier, M., & Tran, U. S. (2019). Which data to meta-analyze, and how? Zeitschrift für Psychologie, 227(1), 64–82. https://doi.org/10.1027/2151-2604/a000357↩︎
The critiques are worth reading in the original: Del Giudice, M., & Gangestad, S. W. (2021). A traveler’s guide to the multiverse. Advances in Methods and Practices in Psychological Science, 4(1). https://doi.org/10.1177/2515245920954925; Modrák, M. (2026). Multiverse analysis, abdication of responsibility and manufacturing of doubt. arXiv. https://doi.org/10.48550/arXiv.2607.14623; Lakens, D., Rasti, S., & Tunç, M. N. (2026). There is only one correct analysis. OSF Preprints.↩︎