Why psychologists must change the way they analyze their data: The case of psi

Overview

It is not itself a report of a séance, an apparition, a spontaneous anomalous experience, or a new psi experiment. Its genre is methodological criticism: it uses the extraordinary implication of a claimed laboratory effect to argue that ordinary psychological research practices and conventional significance testing can produce conclusions that appear stronger than the underlying evidence warrants. The core historical importance of the piece lies in its role as a visible early case study in the wider reform debate over reproducibility, selective analysis, evidential thresholds, Bayesian inference, preregistration, and replication.

The recalled dispute concerns how to interpret nominally significant results from experiments in which participants' responses were said to anticipate future random events or future exposure conditions. Wagenmakers, Ruud Wetzels, Denny Borsboom, and Han L. J. van der Maas reportedly argued that reliance on conventional null-hypothesis significance testing, including one-sided testing and procedures vulnerable to optional stopping or other analytic flexibility, was unsuitable for drawing a dramatic conclusion about psi. Their alternative Bayesian framing was intended to compare support for a null model with support for an effect model, rather than treating a p-value below a conventional threshold as direct evidence for precognition.

This did not constitute an empirical demonstration that no anomalous effect could ever occur. It was an argument about what the available analysis could justify under stated models, assumptions, and prior expectations. The controversy therefore has two separable objects: Bem's empirical claims and the Wagenmakers group's critique of the inferential route from data to those claims.

Keeping them separate matters. A methodological comment can expose weaknesses in an evidential framework without proving fraud, error, or a particular mundane mechanism in every original trial. Conversely, an advocate may dispute the critique's priors, test choices, or characterization of the original analyses without thereby establishing precognition.

Later replications and reform-oriented discussions broadened the case beyond psi, making it a recurrent teaching example about how low-probability claims interact with small effects, researcher degrees of freedom, publication selection, and prior plausibility. The setting is both local and transnational. University of Amsterdam affiliation supplied an identifiable institutional setting for several critics, but the substantive arena was a psychology journal, professional correspondence, conference and classroom discussion, online circulation, and the larger international reproducibility movement.

Reported phenomena in this dossier are thus chiefly statistical and behavioral rather than sensory: participants making choices, estimating preferences, or responding to stimuli in laboratory tasks; analysts selecting directional tests and stopping rules; readers encountering striking claims through abstracts, press coverage, and debate. Sensory descriptions belong only indirectly to the experimental stimuli described in the underlying psi literature, such as emotionally valenced images or erotic material, and should not be confused with independently verified anomalous perceptions. Commercial and reputational factors are relevant as context, not as proof of improper motive.

A counterintuitive claim about predicting the future is unusually attractive to journalists, publishers, lecturers, skeptics, proponents, and online audiences. Journal prestige, citations, professional standing, grants, book sales, paid speaking, and the wider attention economy can reward both spectacular claims and forceful rebuttals. Those conditions may shape which studies, arguments, and retellings travel furthest, while offering no basis by themselves to infer that a named author manipulated data or argued in bad faith.

The lasting cross-case motifs are extraordinary-claim thresholds, flexible analyses, directional testing, Bayesian versus frequentist evidence, selective publication, failed or mixed replication, and the difference between a critique of method and a disproof of a phenomenon.

Chronology

The immediate background was Bem's 2011 publication of laboratory experiments framed as tests of whether present responses could be influenced by future events. Recalled descriptions characterize the results as statistically significant by conventional standards and as involving familiar social-psychological task formats adapted to a retroactive or precognitive interpretation. The sensational implication made the paper an unusually effective test case for arguments about evidential standards.

Later in 2011, Wagenmakers, Wetzels, Borsboom, and van der Maas published or circulated their methodological response under the present title. Their intervention reportedly used Bayesian reanalysis and criticism of inferential practices to contend that conventional significance results did not provide an adequate basis for the broad psi conclusion. Exact sequence, venue details, and any revision history require documentary checking.

The exchange became part of a broader early-2010s discussion of replication and research reform. Subsequent replications, meta-analytic arguments, preregistration initiatives, and debates over Bayesian methods repeatedly cited the case, though those later developments should not be collapsed into a single decisive follow-up experiment. The historical record contains multiple lines of work with differing protocols and standards.

In later retellings, the affair was often reduced to a simple slogan that a psi paper either proved the failure of psychology or was refuted by Bayesian statistics. Both compressions obscure the actual disagreement. The case remains useful precisely because it displays competing assumptions about design, stopping, prior plausibility, error rates, and the meaning of replicated or nonreplicated results.

People, organisations, and setting

Eric-Jan Wagenmakers is recalled as the lead author and a methodological critic associated with the University of Amsterdam. Ruud Wetzels, Denny Borsboom, and Han L. J. van der Maas were coauthors in the recalled lead. Their roles were those of commentators and reanalysts rather than investigators reporting a fresh psi dataset in this particular article.

Daryl J. Bem was the author of the target psi paper commonly known as Feeling the Future. In this dossier, Bem is a central disputant because the critique's arguments depend on how his experiments, tests, and inferential choices are characterized. The critique should not be treated as a finding that he committed misconduct, because methodological disagreement and misconduct are distinct propositions.

The University of Amsterdam, Netherlands, is the supplied location label and should be understood as an affiliation-based setting rather than necessarily the sole site of publication, debate, data collection, or subsequent replication. The relevant social setting was mainstream academic psychology, with statistical methodology functioning as the contested terrain. Journal editors, peer reviewers, readers, replication teams, statisticians, and science journalists all form part of the wider institutional environment.

The principal physical setting of the underlying claims was the psychology laboratory described by Bem, where participants reportedly made responses to computer-mediated tasks. The critique moved that setting into an analytic one: tables of results, assumptions about stopping and tail direction, modeled hypotheses, and debates over evidential interpretation. This transition from laboratory event to statistical narrative is central to the subject.

Reported sensory and behavioral phenomena

The reported behavioral phenomenon under dispute was not a dramatic visible manifestation but a small statistical departure from chance in tasks interpreted as allowing response before a relevant future event. Depending on the task, participants were reported to choose, classify, predict, or react in ways that supposedly aligned with subsequently determined stimuli or outcomes. The intended anomalous interpretation was backward causation or precognition, not ordinary conscious forecasting based on disclosed cues.

Recalled descriptions of the underlying experiments include computer-presented material that could be emotionally charged, neutral, or sexually salient. Participants' ordinary sensory experience would therefore have involved viewing images or instructions on a screen, making keyboard or mouse responses, and perhaps anticipating rewards or emotionally striking content. Such exposure is mundane laboratory stimulation; it is not independent confirmation that a participant sensed a future event.

At the analytic level, the salient reported pattern was a distribution of scores or response times yielding low p-values under selected null models. Critics focused on behavioral traces aggregated across trials and participants, rather than witness testimony about subjective psychic sensations. No stable sensory signature, unique physiological marker, or universally observed subjective state is established by the recalled controversy.

The critique itself reported a different kind of phenomenon: apparent evidential inflation when an extraordinary hypothesis is evaluated with permissive thresholds, directional tests, flexible stopping, or post hoc analytic discretion. That is a methodological pattern, not a paranormal occurrence. Its relevance extends to many ordinary research domains in which effects are small, measurements noisy, and researchers have multiple defensible choices.

Investigation and analytical history

The Wagenmakers group reportedly investigated the published evidential claim through reanalysis rather than field investigation. Their approach contrasted traditional significance testing with Bayesian hypothesis comparison, asking whether the observed data altered relative support for a null account and an effect account. Such calculations necessarily depend on model form and prior distributions, so their force rests on transparent assumptions rather than on an assumption-free statistical verdict.

The criticism reportedly emphasized one-sided tests, conventional thresholding, and optional stopping or stopping-rule vulnerabilities. A one-sided test can be defensible when its directional prediction was specified in advance, but it can appear overly favorable if direction is selected after results are seen. Optional stopping can distort a conventional p-value interpretation when data collection is continued, paused, or ended in response to results without an analysis method that properly accounts for that rule.

The broader investigation also concerns what publication alone can reveal. A finished article may not fully display abandoned conditions, unpublished studies, alternative data exclusions, pilot work, software settings, or all analytic branches considered during research. This absence does not prove that any undisclosed flexibility occurred in a particular case.

It explains why later reform practices favor prospective protocols, data sharing where ethical, code sharing, registered analysis plans, and clearly reported stopping rules.

Replication attempts are relevant corroborative context but are not a uniform evidential object. They may differ from the original in participant pools, materials, randomization, experimenter expectations, exclusions, power, task implementation, and whether the design was preregistered. A fair assessment therefore asks which original claim is being repeated, how closely the protocol matches it, and how divergent outcomes are integrated, rather than treating any single later result as an automatic final ruling.

Disputes and alternative explanations

The main disagreement is epistemic rather than merely ideological. Skeptics emphasize that extraordinary claims require evidence strong enough to overcome substantial prior doubt, and that a p-value is not the posterior probability that psi is true. Proponents or methodological critics of the critique may answer that priors should not be fixed by skeptical intuition, that Bayesian conclusions are sensitive to prior specifications, or that well-designed experimental results deserve consideration without an insurmountable prior penalty.

A second dispute concerns how accurately the comment represents the original statistical procedures. Recalled leads identify one-sided tests, optional stopping, and conventional significance thresholds as central issues, but the exact practices, disclosed rationales, and applicable assumptions must be checked against the primary papers and replies. It would be erroneous to convert a concern about the susceptibility of a method into an unverified allegation about a specific hidden decision.

Mundane explanations for an apparent psi result include sampling variation, multiple testing, selective reporting, outcome switching, weak randomization, unnoticed procedural cues, data-handling mistakes, expectancy effects, and publication bias. These explanations differ in seriousness and likelihood, and none can be assigned to an individual experiment without relevant records. The critique's general logic is that ordinary statistical and procedural mechanisms can generate surprising-looking outcomes more readily than an immediate acceptance of retrocausation would imply.

The counterposition is not necessarily that every positive result is paranormal. It may be that the methods did not exclude a weak, anomalous effect, or that existing models fail to capture the evidence. The decisive issue remains whether the evidence is robust under prospective, transparent, independently repeated designs and whether it discriminates psi from mundane alternatives.

The recalled dossier contains no verified basis for saying that this threshold has been met.

Transmission, retelling, and commercial context

The case traveled well because its title made a general claim about psychology while its example concerned psi. That pairing allowed methodologists to discuss abstract statistical questions through a culturally vivid claim about foreseeing the future. It also allowed paranormal advocates and skeptics to recast the exchange for audiences with different commitments, often emphasizing either the alleged anomaly or the warning about research practice.

Academic transmission likely included citations, seminars, methods courses, journal correspondence, statistical software demonstrations, and later reproducibility discussions. Popular transmission likely included news stories, blogs, podcasts, social media posts, and simplified classroom anecdotes. These routes favor concise conflict narratives, whereas the underlying argument involves details of prior choices, model assumptions, experimental protocol, and the difference between absence of evidence and evidence favoring a null model.

Commercial incentives should be handled cautiously. Sensational claims can generate attention and marketable content, while high-profile critiques can generate professional visibility, citations, invitations, and demand for methodological training or software. Publishers and media outlets may also prefer controversy with clear heroes and villains.

These incentives help explain amplification, but they do not establish personal financial motivation, data fraud, or the correctness of either statistical position.

Later retellings sometimes position the episode as a founding parable of the replication crisis. That phrase is useful only if qualified: the case was one influential public flashpoint among many, and the reform movement had multiple antecedents. Its pedagogical afterlife can obscure the fact that Bayesian and frequentist methods each require judgment, disclosure, and an explicit connection between scientific question and statistical model.

Cross-case connections

The strongest comparison motif is the extraordinary-claim threshold. Cases involving telepathy, presentiment, remote viewing, anomalous healing, or other low-prior laboratory claims often face the same question: how much and what kind of evidence would distinguish a real effect from the accumulated consequences of many opportunities for chance findings. This is a shared methodological structure, not evidence that all such claims have the same cause.

A second motif is analytic flexibility. Small effects can be magnified or made unstable when researchers can choose exclusions, transformations, covariates, outcome measures, stopping points, and tail directions after looking at data. Similar concerns appear in medicine, neuroscience, behavioral economics, and ecology, so psi is a vivid example rather than a uniquely defective field.

A third motif is adversarial transmission. Where a claim touches fundamental beliefs, participants may read null results as suppression and positive results as vindication, while critics may mistake methodological vulnerability for total empirical closure. Cross-case comparison should record this polarization as a social dynamic and separately assess protocols, raw results, incentives, and independent replication.

A fourth motif is the shift from retrospective persuasion to prospective constraint. Preregistration, blinded procedures, fixed stopping rules, independent randomization audits, registered reports, and shared code aim to make later explanations of a result less dependent on post-result choice. These safeguards improve interpretability but do not guarantee truth, and their adoption varies by study and period.

Limits of this recalled synthesis

This dossier is a recalled synthesis built from the supplied lead and model knowledge, not a documentary reconstruction. It has not consulted the named article, Bem's original paper, correspondence, data files, journal records, or later replication reports. Titles, affiliations, broad positions, and chronology are plausible recalled leads that require checking before use as citation-grade facts.

The supplied lead directs attention to optional stopping, one-sided tests, and conventional thresholds, but this dossier does not determine the exact numerical analyses, priors, Bayes factors, sample sizes, or every claim made in the exchange. It intentionally avoids invented quotations, page numbers, archive identifiers, publication details, and URLs. Any detailed assessment should obtain the primary texts and compare criticism with the authors' responses.

The dossier also cannot infer private intentions. Attention, career incentives, publication dynamics, and commercial media interest are structural influences that may shape dissemination. They are not evidence that any named author knowingly exaggerated a result, concealed data, or acted for profit.

Such claims would require specific, independently checked documentation.

Finally, a statistical critique is neither a direct observation of psi nor a conclusive metaphysical proof of its impossibility. Its proper evidential role is narrower and valuable: it identifies what analytical conditions must be met before a surprising laboratory pattern can bear the interpretive weight placed on it. This distinction should govern future comparison and any subsequent source verification.

Chronology

People and roles

Connections to explore

Unretrieved reference leads

1. Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi.

   Eric-Jan Wagenmakers, Ruud Wetzels, Denny Borsboom, and Han L. J. van der Maas. · Suggested not retrieved scholarly methodological commentary.

   This is the supplied recalled lead and is the primary suggested source for checking the critique's precise statistical claims, scope, and assumptions.

   Suggested search: Wagenmakers Wetzels Borsboom van der Maas 2011 Why psychologists must change the way they analyze their data psi.

2. Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect.

   Daryl J. Bem. · Suggested not retrieved empirical article.

   This is the target paper needed to compare the critique's characterization with the original experimental design, tests, and stated conclusions.

   Suggested search: Daryl Bem Feeling the Future experimental evidence anomalous retroactive influences cognition affect.

3. Responses or commentary concerning the Wagenmakers critique and Bem's psi paper.

   Relevant authors, editors, and methodological commentators. · Suggested not retrieved debate record.

   Replies are needed to identify contested assumptions about priors, one-sided testing, stopping rules, and the meaning of Bayesian evidence.

   Suggested search: Bem response Wagenmakers Wetzels Borsboom van der Maas case of psi Bayesian critique.

4. Later preregistered replication studies of Bem-style precognition tasks.

   Independent replication researchers. · Suggested not retrieved replication literature.

   These studies can clarify how later prospective constraints affected outcomes, while preserving differences among protocols and analytic plans.

   Suggested search: preregistered replication Bem precognition Feeling the Future experiments.