In the Reasoner: Study 329

The Reasoner is a monthly digest highlighting exciting new research on reasoning, inference and method broadly construed. It is interdisciplinary, covering research in, e.g., philosophy, logic, AI, statistics, cognitive science, law, psychology, mathematics and the sciences. Each month, there is a column on Evidence-Based Medicine. Here is this month’s column:

Study 329  has  gone  down  in  history  as  one  of  the  most  infamous  clinical  trials  in medicine.   The  study  was  a  double-blinded  randomized  controlled  trial  testing paroxetine  and imipramine  against  placebo  in  adolescents  diagnosed  with major depression. The  conclusion  of  the  study  was  that ‘[p]aroxetine is generally well tolerated and effective for major  depression  in  adolescents’.    Soon  after,  on  the  basis of this study, paroxetine, a selective serotonin reuptake inhibitor, was widely prescribed by doctors for off-label use in children. (Some of the figures are given in this Medicines and Healthcare Products Regulatory Agency (MHRA) report.)

The  study  has  become  infamous  because  the  MHRA  arrived  at  the  opposite conclusion  when  they  later  took  a  look at the study.  The MHRA concluded that the study had failed to demonstrate the effectiveness of paroxetine for treating depression, and in fact demonstrated an increased risk in suicidal ideation and self harm in teenagers. Given this, it was recommended that doctors stop prescribing the drug to adolescents.

Although the drug looked to perform well in terms of a number of outcomes, these were not among the eight outcomes that the study had initially intended to measure.  In fact, the drug performed poorly in terms of the initially specified eight outcomes.   This  is  an instance  of  outcome  switching,  which  is problematic because there is a possibility that a correlation between the drug and an outcome is simply the result of chance. Given this, choosing which outcomes to measure after the fact makes it possible to pretend that a fluke is a significant result. But switching outcomes need not be a bad thing, so long as the switch is pointed out—at least, that is the recommendation in  the  CONSORT guidelines for  reporting  randomized  controlled trials.  The problem with Study 329 was that it looked designed to mislead in this respect.   In part as a response to Study  329,  there began  an  initiative to restore invisible and abandoned trials (RIAT). In particular, there has been a reanalysis of Study 329.  More information is available at Restoring Study 329.

Recently, some results on the prevalence of outcome switching have been published.  And a group at the Centre for Evidence Based Medicine at the University of Oxford has a strategy to remedy this state of a airs.  The COMPare team have begun  systematically checking  trial  results  published  in  the top five medical journals for evidence of undeclared outcome switching.   The  group’s  exact  methods  are  given  here.   The methods involve comparing the outcomes of the published trial results with a trial registry or the trial protocol. In cases where they find a discrepancy, they send a letter to the journal in question pointing out the discrepancy,  recommending that the instance of outcome switching be made clear.

The group have found outcome switching in the vast majority of the trials recently published in the top five medical journals. (The results so far are listed here.) They have received some interesting and varied responses from the journals that they have contacted,  and they are listing these responses on their blog. The responses have varied from an acknowledgement and correction of the instances of outcome switching to an unwillingness to publish the group’s letters. In an interview to Retraction Watch, the project members have said:

Until  we  began  writing  to  journals,  we  only  knew that outcome switching was highly prevalent, despite most  journals  promising  to  adhere  to  high  reporting standards.  Now, from the responses we’ve had,we’re learning why it continues to be so prevalent, we are identifying the recurring misunderstandings and systemic  shortcomings.   Essentially  we’ve  solicited qualitative data on the reasons why outcome switching occurs in journals, and it could only have been done by writing these letters.

The group are now writing up a paper with an analysis of these results. But until then, readers can follow the action over at the COMPare blog.

Image Credit: By Ryan (Flickr) [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons

Leave a Reply

Your email address will not be published. Required fields are marked *