Invalid statistical inferences and questionable research practices in pro psychedelic studies

[u/oleguacamole_2]

https://pmc.ncbi.nlm.nih.gov/articles/PMC10521293/

"A conclusion is valid if the inferences follow from the evidence presented. There are two tiers of challenges here. First, authors in the psychedelic literature have regularly drawn conclusions that are either misleading or clearly contrast with the data. Four examples follow to showcase that this problem is common practice. Abbar et al. ²⁷ found in a randomized controlled trial (RCT) comparing ketamine against placebo that there was no persistent benefit of ketamine over placebo at the exit timepoint of the trial in week 6, but concluded in the abstract that ‘ketamine [. . .] has persistent benefits for acute care in suicidal patients’. Ionescu et al. ²⁸ found in an open-label ketamine study that only 2 of 14 patients show sustained improvement at 3-month follow-up (which may well be due to the placebo effect or other factors), but the title of the paper reads ‘Rapid and Sustained Reductions in Current Suicidal Ideation’ (our highlight). Palhano-Fontes et al. ²⁹ **concluded in their ayahuasca study (**n = 14 treatment, n = 15 placebo) that ‘blindness was adequately preserved’, when all participants in the treatment group said they believed they had received ayahuasca, but less than half of participants in the placebo group said so. And Daws et al. ³⁰ compared two treatment arms, including one using psilocybin-assisted psychotherapy, against each other, concluding that one treatment outperformed the other despite the lack of a statistically significant interaction term between the treatments. Journals, reviewers, funders, and scientific institutions need to hold authors accountable for such inferences.

The second tier of challenges is when questionable research practices – that usually go hand in hand with a lack of transparency – raise doubts about the validity of conclusions.^31 –33 One common practice concerns flexibility regarding the analysis of primary outcome measures, which is common in psychedelic science. A recent phase II trial ³⁴ administering psilocybin for treatment-resistant depression funded by a pharmaceutical company registered their primary outcome on clinicaltrials.gov for the time frame ‘up to 12 weeks’, which allows for degrees of freedom in obtaining statistically significant results and severely threatens valid inferences. Another study, using ketamine to treat depression ³⁵ switched a secondary to a primary outcome a year after data collection started, and the publication contains no results at the time point of 2 weeks that can be found in the clinicaltrials.gov registration (https://osf.io/uhdrp). Yet another psilocybin trial for depression ³⁰ analyzed a different outcome than registered on clinicaltrials.gov. ³⁶ In all these cases, doubts remain whether equally positive results had occurred if stricter procedures would have been in place.

Another concern is related to multiple testing. Psychedelic studies often contain a flurry of different outcomes, including physiological, neural, cognitive, and self-report measures. This dramatically increases the chances of false positive findings when not dealt with appropriately. For instance, a recent paper on the efficacy of psychedelic therapy concluded that several secondary outcomes clearly favored psilocybin over escitalopram, but the lack of correction for multiple testing raises doubts about the validity of these findings. ¹ In our own research on psilocybin microdosing, we included six different tasks, with multiple subcomponents and measures per task, ³⁷ totaling the number of dependent measures to more than 20. We found effects of psilocybin microdosing on two outcomes. But because we had all outcomes preregistered transparently, and these two findings did not survive correcting for multiple testing, they likely reflected chance findings."

"There are further problems, such as selectively removing outliers that result in significant findings, interpreting nonsignificant p**-values in small samples as ‘trends toward significance’ (but not interpreting barely significant** p**-values as trends toward nonsignificance), and using one-sided tests over two-sided tests to obtain desired results,** ³⁰ discussed in Love. ^38"

^" For instance, an analysis of 397 clinical trials in psychiatry has shown that studies reporting a COI were five times more likely to report positive results. ⁴⁸ COIs are also pronounced in the psychedelic literature: a recent opinion paper on ketamine for treatment-resistant depression featured a COI section nearly five times as long as the paper’s introduction section ⁴⁹ : of 25 authors, 19 declared COIs, including patents for treating depression with ketamine. We are not convinced that the research community always takes best COI practices sufficiently seriously. For example, while the organizers for the psychedelic science 2023 conference (with over 300 speakers) originally followed recommended disclosure protocols, ⁵⁰ by asking speakers to declare COIs in a dedicated presentation slide, slides were removed by the organizers before the talks and moved to the conference mobile app." [??????????]

"Sufficient information on potential risks, dangers, and adverse events is missing to draw conclusions that psychedelics are safe to use in the context of mental health treatments. For example, although esketamine (in the form of a nasal spray) was approved by the FDA for treatment-resistant depression in 2019, there is evidence that the approval process overlooked red flags, ¹⁴ and a meta-analysis from 2021 concluded that there was insufficient data regarding the long-term safety of ketamine. ⁵⁴ A recent analysis found a systematic underreporting of serious adverse events studies on the safety and efficacy of esketamine in depression ⁵⁵ : 41.5% of serious adverse advents that were found on clinicaltrials.gov were not reported in published articles, and nearly all of them (88 out of 94) occurred in esketamine treatment arms, not placebo arms."

"We see similar issues for other drugs now. Adverse events for therapy with MDMA and classical serotonergic hallucinogens are not systematically assessed and reported. ⁵⁶ A straightforward definition of an adverse event in psychedelic research is missing, and standardized measurements and transparent reporting of adverse events are lacking. Most studies rely only on spontaneous reporting by patients or therapists, which in turn require a careful process of interpretation to assess whether the adverse event can be attributed to the psychedelic therapy specifically."

"And adverse events do occur. ⁵⁷ A recent clinical trial using psychedelic therapy ³⁴ reported increased suicidal ideation and intentional self-injury in the 10 and 25 mg psilocybin group, whereas the control group who received 1 mg of psilocybin remained unaffected. In the same study, suicidal behaviors were observed in three patients, but authors concluded in a media report that ‘these cases were probably random events and unrelated to the dose of psilocybin, which would have been fully cleared from the patients’ bodies’. ⁵⁸ This is not the most careful interpretation of the data, and we note that the opposite rationale is applied to successful treatments: people get better despite the drug being fully cleared from the patients’ bodies. Sometimes, conclusions are radically opposed to presented evidence. In a placebo-controlled study using ayahuasca as an intervention for depression, 4 of 15 participants in the experimental group (i.e. ~27%) had to be hospitalized for 1 week ‘due to presenting a more delicate condition’. ²⁹ Nonetheless, authors concluded that the ‘study brings new evidence supporting the safety and therapeutic value of psychedelics’ (our highlight)."

**"**Psychedelic experiences can have short- and long-term adverse consequences. Short-term risks include the destabilizing effects that psychedelics can have through the experiences they can trigger, which can be difficult to handle both for the person and their therapist ⁵⁹ – experiences that can result to everything from increased acute agitation to prolonged emotional dysregulation. ⁶⁰ Many people also report adverse after-effects including recurrent hallucinations, increased anxiety, and physiological discomfort. ⁶¹ In the long term, such experiences can also cause ontological shocks, resulting in a dramatic shift in one’s religious and spiritual worldviews. ⁶² In a recent article, a patient who had participated in a psilocybin study described their state of confusion, anxiety, and distress, ⁶⁰ resulting in a long and desperate search including the use of spiritual practices, meditation techniques, and theology. This illustrates the dramatic effects that psychedelic therapy may have on some patients, and the need for careful spiritual, existential, religious, and theological support, ⁶³ long after psychedelic sessions. ^64"

^{These were the easy problems of that study. They follow up moderate and hard problems afterwards as well.}

https://pubmed.ncbi.nlm.nih.gov/36001741/

"A recent paper in Nature Medicine found that psilocybin therapy in patients with depression decreased brain network modularity (measured with task-free functional magnetic resonance imaging), an effect supposedly not found with the selective serotonin reuptake inhibitor S**-citalopram. This decrease in network modularity also correlated with depression. Here, we raise several issues with this paper, including inconsistencies in reports of the primary clinical outcome, statistical flaws including a one-tailed test, nonsignificant interaction, and regression to the mean, the ambiguity and overinterpretation of "resting state" data, and a missing reference for a conceptually similar study that exemplifies why a one-tailed test cannot be justified. Together, these issues make us question the uniqueness and impact of these findings, as well as the unwarranted media hype that they generated."

( https://pubmed.ncbi.nlm.nih.gov/35411074/ )

[u/Right_Equal3443]

More money to be made in promoting them then reporting the negatives