In 2017, I published a study in Personality and Individual Differences with Steven G. Ludeke. Our motivation for conducting the study was that other studies uncritically used the Big Five data in the World Values Survey without evaluating the reliability of the data.
In brief, and to recap, the data was unable to capture inter-individual variation in Big Five personality traits and should be used with caution. Specifically, we showed that the distribution of item-item correlations for the Big Five personality traits were unsatisfactory:
The main reason we decided to write up the short paper was that we could see that different researchers published studies using this data. Accordingly, we hoped that people would read our study and not use the data (and thereby not cite our study).
Of course, some of the citations to our study point to the challenges with measuring Big Five traits. For example, in a study by Laajaj et al. (2019), that I have commented on in the New Scientist, the authors simply point out what we find:
Related, Ludeke and Larson [sic.] (29) flag concerns with the use of the BFI-10 (30), a short 10-item Big Five instrument used in the World Values Survey, showing low correlations between items meant to measure the same PT.
We are happy to see researchers pay attention to our study and share the concern. However, what I have noticed – and what I am concerned we will see more of in the future (and the reason I write this post) – is that some researchers continue to use the data even when they are aware of the limitations/problems.
The most recent example is this study. Here is a brief description from the abstract on what they do: “Using the most recent wave of the World Values Survey, this study investigates the impact of personality on individual protest participation in 20 countries using the multilevel modelling.”
Importantly, the authors do nothing to take the problems with the data into account. Furthermore, they are aware of the problems with the data. As they write in a footnote:
A recent study done by Ludeke and Larsen (2017) points out the problems with the Big Five assessment in the WVS. However, they are not able to come up with any solution to the data challenges posed to the WVS. Given data availability, the WVS is the only choice to conduct cross-national comparative research on personality.
I disagree very much with the reasoning in the footnote.
First, the fact that we do not come up with a solution is a major red flag. People have looked into what can explain the low reliability and, similar to us, have been unable to find a solution (see, for example, this post by Rene Bekkers). Since we do not know what causes the problem, we do not have a solution that can make the data useful. Accordingly, I am not sure I understand the meaning of “however” in the argument, implying that there is a problem but it’s not really a problem. Until somebody can offer a solution (if such a solution exists), I highly recommend that you do not use the Big Five measures in the World Values Survey.
Second, the World Values Survey is not “the only choice to conduct cross-national comparative research on personality.” For example, the 2010 wave of the Latin American Public Opinion Project includes data on Big Five traits in a comparative setting (we also use this data in our study in the European Journal of Personality). There are also several studies using Big Five traits in a comparative perspective (e.g. Curtis and Nielsen 2018, Gravelle et al. 2020 and Vecchione et al. 2011). In other words, the World Values Survey is clearly not the “only choice to conduct cross-national comparative research on personality”. My recommendation is that if you have no choice but to use the Big Five data in the World Values Survey, limit the data to the Netherlands and Germany (where the reliability measures are satisfactory).
My general recommendation: Stop using the Big Five measures in the World Values Survey – even if it’s a good opportunity to cite our critique (seriously, I couldn’t care less about the citations).