In a new study, Adults who microdose psychedelics report health related motivations and lower levels of anxiety and depression compared to non‐microdosers, a team of researchers conclude that microdosers, i.e., people who use psychedelic substances at sub‐sensorium ‘microdoses’, exhibit lower levels of depression, anxiety, and stress.
The study relies on self-reported, observational data. As this is not a placebo-controlled study that could establish a strong conterfactual group, it is important to examine how exactly the study constructs a treatment group of microdosers and a control group of non-microdosers. In this post, I will show that there are some significant limitations, maybe even problems, with the study design and inferences made in the paper.
Figure 1 in the paper provides a flow chart with the steps used to construct the two groups of interest. The baseline survey was completed by 8,703 respondents (from 84 nations), as 20% of the respondents did not provide sufficient data to take part in the study. Around 52% of the respondents who entered the study also filled out the Depression, Anxiety, Stress Scale-21 (DASS-21), although the manuscript is not providing any information on how these 52% of the respondents differ from the people participating in the study not answering these questions.
Well, let’s begin from the … beginning. How did people find their way into the study? Is it a random sample? No, on the contrary, people self-selected into the study, so do not assume that any conclusions made in the study will generalize to the population at large (whatever that population might or might not be). According to the webpage of the study, the study was launched on November 15, 2019, on the Joe Rogan Experience podcast. I am quite confident concluding that people not only listening to Joe Rogan, but also deciding to participate in a study on microdosing, are weird (not psychologically WEIRD, but weird-weird). Furthermore, as the authors acknowledge in the paper, only iPhone users could participate in the study (there was no Android app).
The next challenge is identifying who the microdosers are. You can see in the figure that the survey asks “are you currently engaging in a practise of microdosing”, and the relevant word here is “currently“. From what I can tell, the authors did not ask about previous microdosing experience. Why is that important? When the control group, i.e., non-microdosers, will include those who have never microdosed and those with a history of microdosing, we are introducing a bias. For example, it might be that those who are no longer microdosing had negative experiences with microdosing and, for that reason, worse mental health. This makes it very difficult to substantiate that any differences between the two groups can be attributed to the, for example, positive effect of microdosing in the treatment group (and not a negative effect of microdosing in the control group).
Unsurprisingly, we see that there are significant diffences between the two groups in terms of who are microdosers and non-microdosers. Table 1, shown below, shows that microdosers tend be older (highlighted with yellow and green) and more likely to live in urban community settings (highlighted with blue and pink). Is that something that is taken into account in any of the statistical models? Alas, no. None of the models even attempt to adjust for any of these covariates. It’s hip to chi-square.
In Table 2 in the paper, shown below, we see that microdosers are less likely to use alcohol frequently and more likely to abstain from tobacco (though more likely to use cannabis more frequently). How can we say that any differences between the two groups can be attributed to microdosing psychedelics and not, say, smoking cannabis? Or not drinking alcohol? We can’t.
More interestingly, microdosers are more likely to report a history of mental health problems. Specifically, microdosers are more likely to report having experiences with depression and PTSD/trauma-related mental health problems.
Well, this is interesting. The study is about the positive impact of microdosing on mental health. Notice this part from the paper: “Microdosers were generally similar to non‐microdosing controls with regard to demographics, but were more likely to report a history of mental health concerns. Among individuals reporting mental health concerns, microdosers exhibited lower levels of depression, anxiety, and stress across gender.” How can the difference between the two groups suddently change sign from negative to positive?
Say hello to our good old friend: conditioning on a collider. We have a sample selection based on values of the collider (mental health) that creates a non-causal association between microdosing and anxiety/depression. People who microdose are more likely to report mental health concerns, and depressed people are more likely to report mental health concerns. When we select observations based upon mental health concerns, we select observations that are more likely to microdose or being depressed (on average). There is simply no justification for conditioning on mental health status (it also reminds me of the birth weight paradox within epidemiological research).
The key findings are reported in Figure 2 in the paper:
Here we see a negative “effect” of microdosing psychedelics on anxiety, depression and stress. Again, we are conditioning on a collider so don’t put too much into the results. Actually, ignore them altogether. However, even if we assume that these results were valid, we can add a few points. First, notice how the figure uses the y-axis to mislead. Second, and related, the “effect” sizes are negligible. Maybe that is why it is called microdosing.
Noteworthy, I have not looked at the raw data so I cannot rule out that I have identified all relevant issues with the empirics. The data availability statement of the paper says: “All data generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.” I reached out to the corresponding author and asked for the replication material. Specifically, I said: “I have a few ideas for robustness tests I would like to explore to better understand the findings, and I am wondering whether you can share the replication data. I can confirm that I will not share the data with anybody else. I hope this constitutes a reasonable request.” The researchers were kind enough to let me know that they are still ‘preparing the data for public release’, but at the time of me writing this post, I cannot analyse the data. I will of course write a follow-up post when the data is available if – and only if – there is anything of interest to say.
I always assume that researchers care about testing a hypothesis rather than looking for support for a hypothesis. However, in the specific case I am doubtful that the researchers have put the theory to a fair test. Why? Take a look at the (depressingly long) competing interests statement in the paper:
Joseph Rootman has received research funding from Quantified Citizen Technologies who provided the data collection platform for this study. Pamela Kryskow is a member of the clinical advisory board of Numinus Wellness which is a company that provides psychedelic psychotherapy services. Pamela Kryskow is compensated for this role with Numinus stock. Kalin Harvey is the CTO and co-founder of Quantified Citizen which is a company that produces software for decentralized mobile research, which was used in this study. Paul Stamets is a minority investor in Quantified Citizen and is an applicant on pending patents combining psilocybin mushrooms, Lions Mane mushrooms and niacin. Eesmyal Santos-Brault is the CEO and co-founder of Quantified Citizen which is a company that produces software for decentralized mobile research, which was used in this study. Kim PC Kuypers is a principle investigator on research projects, the present study not included, that are sponsored by Mindmed and Silopharma which are companies that are developing psychedelic medicines. Vince Polito is a science advisor for Mydecine Innovations Group, which is a company that is developing psychedelic medicines. Francoise Bourzat is a collaborator in the study on psilocybin assisted psychotherapy for COVID related grief at Pacific Neuroscience Institute, Santa Monica, CA. Zach Walsh is in paid advisory relationships with Numinus Wellness and Entheo Tech Biomedical regarding the medical development of psychedelics.
We have both a theoretical bias (these researchers are being paid to support the medical development of psychedelics) and a methodological bias (using and showing the validity of a specific data collection platform). The competing interests statement in the paper can easily explain why and how the researchers can end up showing a negative correlation between microdosing psychedelics and anxiety, depresion and stress, even when the raw data shows positive correlations.
Will microdosing psychedelics have an impact on your mental health? If so, is it positive? Or negative? We cannot use any of the findings in the paper to answer these (relevant) questions. Hopefully future research can help us out here.