It is getting increasingly popular within the social sciences to preregister studies. That is, prior to collecting any data for a study, researchers register their theoretical predictions/expectations (i.e., hypothesis or hypotheses), the data collection procedure, the planned analysis, etc. I am generally in favour of preregistrations. All else equal, more information on how a study was conducted is better, and it is valuable to be able to compare differences between a final paper and the preregistration plan.
It is no surprise that preregistrations are popular, and maybe even needed. We are all too familiar with serious challenges in empirical work – p-hacking, HARKing, low statistical power, etc. However, my concern is that preregistrations are not sufficient (or even necessary) in order to address some of these challenges, and they can, at worst, make us more likely to believe that such challenges are no longer present.
I have reviewed a few papers for different journals where the authors have preregistered their study (or studies). It is great to see researchers develop preanalyses plans and preregister them, and it is really helpful as a reviewer to be able to check what the researchers planned in relation to a specific study. However, more often than not, I am not convinced that the preregistration is adding as much as we are led to believe.
A specific concern is that we are currently promoting a culture where preregistered studies are not only more robust than non-preregistered studies, but also more valid and less in search of replications. A preregistration will not cover all choices researchers can make, and preregistrations might give a false belief that the results are stronger and more robust than a replication will show.
I have previously written about how even a preregistration is leaving a lot of flexibility in how researchers will analyse and write up their studies. (See, for an example, this post I wrote last year on whether wolf attacks predict far-right voting.) If you include a few hypotheses and measures in a preregistration, you can simply select the features in the preregistration that will lead to the preferred effects. One could say that this is a feature of preregistrations, i.e., that we can easily spot such issues, but this will require that reviewers and researchers are actually able to critically evaluate and reflect upon such limitations and challenges.
However, there are more important limitations with preregistrations. Researchers can decide whether a study is relevant (e.g., publishable) once they have conducted it, decide whether a study was preregistered once the study was conducted and analysed, make multiple preregistrations before conducting a study, etc. In other words, we have no overview of when researchers make preregistrations and how many they make, and what specific data collection endeavours they are linked to.
Consider a hypothetical example. If I got funding to explore the political correlates of Big Five personality traits in a nationally representative survey, I could come up with ideas for three or more papers (i.e., multiple papers based upon the same survey). I could then write a preregistration for each of the three papers with five different hypotheses in each (i.e., a total of 15 different hypotheses). Once I have collected and analysed the data, I could then 1) focus on the paper with the most impressive results (and ignore the two other papers) and 2) pay more attention to the hypothesis that showed the strongest result. The preregistration for the paper would allow other researchers to discuss the limitation of focusing on only a few of the hypotheses in the preregistration, but never the fact that there were two papers that I decided to not write.
One of the reasons why I believe this hypothetical example is relevant is the fact that preregistrations can stay private until researchers decide to announce that a study was conducted and a paper using the study was submitted to a journal. AsPredicted, for example, is explicit about this limitation: “Requiring that all pre-registrations (eventually) become public makes pre-registering less appealing to authors. In an environment where authors need to voluntarily pre-register, this reduces the number of studies that will be pre-registered. That’s the cost.”
I am not sure why requiring that all preregistrations (eventually) become public will make preregistering less appealing to authors. You cannot have your cake and eat it too. On the contrary, it should make preregistering more appealing to authors and the scientific community.
We know that researchers often decide to only write up the most promising results (i.e., the studies that can end up in the best journals). For that reason, it is no surprise that a lot of preregistrations will never end up in the public domain linked to an actual paper. And I believe the current norm within the social sciences is to not be transparent about such decisions.
There are exceptions where researchers are explicit about the procedure where specific preregistrations did not end up in a paper. Consider for example a study in Journal of Experimental Social Psychology on how dominant groups support digressive victimhood claims to counter accusations of discrimination. In the OSF folder for this study, you will find a note on the preregistered studies. Here is the introduction in the note:
In our manuscript, we report three studies. In this OSF folder, we report four preregistrations, two for Study 2 and 3 as they appear in the paper, and two for studies that we do not report in this paper. Below we openly describe potential limitations of our preregistration process and our reasoning for not including our second preregistered study in our paper. Implications for the robustness of our findings are discussed.
The authors made preregistrations for studies that did not end up in the paper. It is great to see the authors being explicit and transparent about this, but this is the exception rather than the norm. For Study 2 in the paper, the authors first describe that the “formal preregistration for Study 2 was not uploaded in advance of data collection or analysis”. Accordingly, due to a mistake, the preregistration was not actually uploaded prior to the study being conducted. If the authors had decided that Study 2 was not relevant for the paper, would they then still have made it public that they made this mistake? I am happy to assume that the specific authors would do that, but I see no reason to give most researchers the benefit of the doubt.
Next, in the discussion of the unreported preregistered studies, the authors discuss how the data and the results made them decide against including those studies in the paper. For the “Unreported Preregistered Study #2”, for example, the authors note that “We believe the timing of this study may have shaped the lack of predicted effects.” as a rationale for not reporting the study. I am not convinced by this reasoning, and I am concerned that most researchers are able to find good reasons for not including a preregistered study if they find a ‘lack of predicted effects’. My specific concern is that we have no overview of the universe of reasons researchers have to dismiss a study (and thereby the preregistration). And, more importantly, researchers will only rely on such reasons once they have examined the data.
If we acknowledge the need for preregistrations, and see them as a necessary condition, or even a sufficient condition, for a robust and reliable science, why should we trust the flexibility researchers have in the process of designing preregistrations? Or, simply put, how can we believe researchers without ‘a preregistration’ of the preregistration(s)?
Again, my concern is that we ignore such limitations and, instead, are much more likely to believe the results of a study just because there is a ‘preregistration’ linked to the study. Accordingly, researchers might be less likely to see the need for a replication of a study if it was preregistered.
For those reasons, I believe it is much more important to focus on what happens after a study is published than what happens before it is conducted. If a study is important, I find it irrelevant whether it was preregistered or not. What I want to see is an independent replication (and preferably multiple independent replications). A strong and important paper merits a replication – not because we have any reason to believe the findings might not replicate, but because we care about whether they do or not. With a lot of social science research, we more often than not do not care about whether the findings replicate. Again, my concern is that we begin to care even less once we devote more attention to preregistrations.
Once a paper is published, the digital object identifier should be seen as the preregistration of the replication study. Specifically, a paper (and the supplementary material) should contain enough information to enable other researchers to replicate the findings. If there is not sufficient information in the paper to provide a fair replication, there is something missing from the paper. Preferably, once a paper is accepted for publication, the authors should complete a postregistration to make it easier for other researchers to replicate the key findings.
In sum, the preregistration of a study should not make us overrate the robustness and credibility of a result, and make us less likely to consider a replication. The published paper should be the preregistration of the replication.