Symposium for Meta-analyses, meta-meta-analyses, and meta-research in individual differences research (DPPD 2019)

Symposium for Meta-analyses, meta-meta-analyses, and meta-research in individual differences research (DPPD 2019)

Looking forward to my talk on differences in intelligence between twins and singletons at the DPPD 2019!

It will be part of the symposium Meta-analyses, meta-meta-analyses, and meta-research in individual differences research, see a complete description of the talks below!

15th Conference on Personality and Psychological Assessment:

The science of personality and psychological assessment is becoming increasingly collaborative. Researchers in the field now routinely share their hypotheses, data and materials with one another, which enables them to pursue their research questions on a larger scale, in a more co-ordinated manner, and with greater transparency.

The 15th biennial DPPD conference will celebrate this and other developments in the spirit of a truly Open Science. Together, we will highlight the progress that has already been made in the field, and identify key issues yet to be resolved. For the first time, the primary conference language at DPPD will be English. Researchers from all over the world are cordially invited to participate and to submit their best work. You can find the full program here.

Symposium: Meta-analyses, meta-meta-analyses, and meta-research in individual differences research:

In the wake of the replicability debate and confidence crisis of the 2010s and the now ensuing renaissance (Nelson, Simmons, & Simonsohn, 2018) in psychological science and other empirical disciplines, a number of important reactions and innovations have gained momentum. These positive trends include replication initiatives of unprecedented scale, a general method reform, the open-science movement, and statistical tools for assessing the evidential value of empirical findings. As well, these events have spurred further meta-analytic method development and applications, including second-order research synthesis (meta-meta-analysis), and have led to the emergence of meta-research (research on research and its practices). This symposium highlights the relevance of these latter three approaches of evidence accumulation for building consilient knowledge in individual differences research. The 4 presentations report research conducted in the context of the Faculty of Psychology, University of Vienna, branch within the Network of Open-Science Initiatives at Psychology Departments (NOSI; and specifically address: (1) a long-standing question in intelligence research (Plessen); (2) effect declines in intelligence research and elsewhere (Pietschnig); (3) publication bias in individual differences research (Siegel); (4) and meta-research on the “Journal of Individual Differences” (Voracek).

1. The conundrum of twin-singleton differences in intelligence: A centenary (1924-2018), preregistered meta-analysis

Constantin Y. Plessen

Twin-singleton differences in IQ would be of concern for many groups, starting with parents of multiples and educators, up to behavioral geneticists (who depend on twin-singleton equivalence for generalizing twin study results to the general population) and cognitive epidemiologists (who investigate IQ associations with morbidity and mortality). It is thus unsurprising that twin-singleton differences in IQ have been investigated since the advent of twin studies and IQ test batteries (Merriman, 1924). We significantly update and expand the sole meta-analysis on this topic (Voracek & Haubner, 2008). Extensive literature search strategies identified 453 eligible effects from 108 studies from 25 countries (totalling 6,100,000+ singletons and 184,000+ twin individuals), published 1924-2018. Multilevel meta-analytic modelling showed twins scored about 4 IQ points lower than singletons: a non-trivial effect, when considering tail ratios and liability-threshold models. Further, the effect was temporally stable, robust with respect to publication bias, generalized to a surprising scope (across sex, geography, measuring instruments, premature birth status, and assisted reproductive technologies), whilst systematic sources accounting for observed cross-study effect heterogeneity remained elusive. The conundrum of this group effect in IQ seems not satisfactorily solvable with the available evidence. Specifically, data from adults and less developed countries remain a desideratum.

2. It’s not getting any better: Systematic, strong, and overproportional effect declines over time are not confined to intelligence research

Jakob Pietschnig

Empirical sciences in general and psychological science in particular are plagued with reproducibility problems. Initial (exploratory) effects often are substantially inflated, hard to replicate, or false. However, systematic empirical accounts on the prevalence, strength, and causes of non-reproducibility are lacking. Here, we show effect declines over time in two meta-meta-analyses. First, we report results based on 22 meta-analyses (k=36; N=697,639) in the journal “Intelligence”. We show that initial reports misestimated summary effects by a small-to-moderate effect size (r = .17). Inflated effects were larger (r misestimations = .18 vs. .08) and twice as likely than deflated effects. Misestimations were positively associated with initial study effects and negatively with initial study sample sizes. Second, we replicated the results in a preregistered synthesis of meta-analyses (k = 247; N = 271,000,000+) from four high-impact journals in psychology. Again, effect declines were more prevalent than increases at a 2:1 ratio and related to initial study effects and sample sizes, but effect misestimations were larger than in intelligence research (r misestimations = .19; .23 vs. .11 for decreases and increases). We argue that the virtually ubiquitous, non-trivial effect misestimations in psychological science are (i) more frequent and stronger for declines than increases, (ii) associated with initial study properties, and (iii) attributable to strategic submission behavior.

3. Publication bias in individual differences research is underestimated: A meta-meta-analysis

Magdalena Siegel

Effect inflation poses a serious threat to the validity of empirical findings and has often been demonstrated to be due to publication bias. Based on data from 42 meta-analyses (2,793 effect sizes, 780,700+ participants) published in “Personality and Social Psychology Review”, we show that publication-bias prevalence in individual differences research is considerably underestimated. In fact, reanalyses of these meta-analyses by applying seven standard (e.g., trim-and-fill) and more modern (e.g., p-curve, p-uniform) methods for publication-bias detection yielded evidence for publication bias in about 37% of cases. This contrasts considerably lower author-reported bias prevalence (about 10%), yielding a ratio of more than 3:1. Moreover, we show that applications of bias assessment in individual differences meta-analyses are comparatively common (i.e., 76% of authors use at least one method; Mdn number of methods = 2, range = 1-8), but mainly rely on subjective (i.e., visual funnel-plot inspection) or outdated methods (i.e., failsafe N). Because different properties of publication-bias detection methods entail differing sensitivity, bias underestimation may be mainly driven by the use of inappropriate methods, in addition to studies that do not assess bias. We argue for the use of (i) several and (ii) state-of-the-art methods of bias detection, as the current overreliance on popular, yet outdated, methods masks bias in individual differences research.

4. Meta-research on the development of the N-pact factor of the “Journal of Individual Differences” (erstwhile “Zeitschrift für Differentielle und Diagnostische Psychologie”), 1980-2017

Martin Voracek

The N-pact factor (NF), defined as the median sample size of studies published in a journal in a year, has recently been proposed as a novel quality attribute of journals (Fraley & Vazire, 2014), alternative to the widely criticized journal impact factor. Initial meta-research investigations into the NF have evidenced NF figures hovering merely about 100, or slightly beyond, for fields like social, sport, and exercise psychology, whereas NFs of journals in clinical psychology and suicidology are appreciably higher (about 180 and 200, respectively). One shared limitation of these accounts is that observation periods throughout were contemporary; hence, historical perspectives on the NF are lacking, and possible time trends in the NF are unexplored. In this first long-term and largest NF analysis to date (based on about 3700 articles; manually coded and evaluated, instead of relying on text mining), we tracked annual NFs of the “Journal of Individual Differences” (the erstwhile “Zeitschrift für Differentielle und Diagnostische Psychologie”) from its inception (1980) up to 2017. We compared the trajectory of this journal’s NF with the one of “Personality and Individual Differences” (an obvious object of comparison, likewise founded in 1980). Inter alia, main results include that the annual NF indeed gradually increased over time and overall amounted to about 190. Discussion focuses on possible causes for this favourable evidence and refreshing development of the journal.

Looking forward to it!