The Reasoner is a monthly digest highlighting exciting new research on reasoning, inference and method broadly construed. It is interdisciplinary, covering research in, e.g., philosophy, logic, AI, statistics, cognitive science, law, psychology, mathematics and the sciences. Each month, there is a column on Evidence-Based Medicine. Here is this month’s column:
Study 329 has gone down in history as one of the most infamous clinical trials in medicine. The study was a double-blinded randomized controlled trial testing paroxetine and imipramine against placebo in adolescents diagnosed with major depression. The conclusion of the study was that ‘[p]aroxetine is generally well tolerated and effective for major depression in adolescents’. Soon after, on the basis of this study, paroxetine, a selective serotonin reuptake inhibitor, was widely prescribed by doctors for off-label use in children. (Some of the figures are given in this Medicines and Healthcare Products Regulatory Agency (MHRA) report.)
The study has become infamous because the MHRA arrived at the opposite conclusion when they later took a look at the study. The MHRA concluded that the study had failed to demonstrate the effectiveness of paroxetine for treating depression, and in fact demonstrated an increased risk in suicidal ideation and self harm in teenagers. Given this, it was recommended that doctors stop prescribing the drug to adolescents.
Although the drug looked to perform well in terms of a number of outcomes, these were not among the eight outcomes that the study had initially intended to measure. In fact, the drug performed poorly in terms of the initially specified eight outcomes. This is an instance of outcome switching, which is problematic because there is a possibility that a correlation between the drug and an outcome is simply the result of chance. Given this, choosing which outcomes to measure after the fact makes it possible to pretend that a fluke is a significant result. But switching outcomes need not be a bad thing, so long as the switch is pointed out—at least, that is the recommendation in the CONSORT guidelines for reporting randomized controlled trials. The problem with Study 329 was that it looked designed to mislead in this respect. In part as a response to Study 329, there began an initiative to restore invisible and abandoned trials (RIAT). In particular, there has been a reanalysis of Study 329. More information is available at Restoring Study 329.
Recently, some results on the prevalence of outcome switching have been published. And a group at the Centre for Evidence Based Medicine at the University of Oxford has a strategy to remedy this state of a airs. The COMPare team have begun systematically checking trial results published in the top five medical journals for evidence of undeclared outcome switching. The group’s exact methods are given here. The methods involve comparing the outcomes of the published trial results with a trial registry or the trial protocol. In cases where they find a discrepancy, they send a letter to the journal in question pointing out the discrepancy, recommending that the instance of outcome switching be made clear.
The group have found outcome switching in the vast majority of the trials recently published in the top five medical journals. (The results so far are listed here.) They have received some interesting and varied responses from the journals that they have contacted, and they are listing these responses on their blog. The responses have varied from an acknowledgement and correction of the instances of outcome switching to an unwillingness to publish the group’s letters. In an interview to Retraction Watch, the project members have said:
Until we began writing to journals, we only knew that outcome switching was highly prevalent, despite most journals promising to adhere to high reporting standards. Now, from the responses we’ve had,we’re learning why it continues to be so prevalent, we are identifying the recurring misunderstandings and systemic shortcomings. Essentially we’ve solicited qualitative data on the reasons why outcome switching occurs in journals, and it could only have been done by writing these letters.
The group are now writing up a paper with an analysis of these results. But until then, readers can follow the action over at the COMPare blog.
Image Credit: By Ryan (Flickr) [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons