According to a report published today at Reuters, antipsychotic drugs appear to be losing their efficacy. This is supposedly because there is a rise in the affect of the placebo effect. The Reuters story is well-written (go and read please), but is missing a “Part II”. If the data are correct and there is a narrowing gap between drug effect and placebo effect, what are the most likely explanations? Is it true that placebo responses are rising? Or that drugs are becoming less effective?
Let’s step back and examine the question being asked, and the possible answers.
When clinical drug trials are done, the drug in question (the test drug) is usually compared to a placebo, an inert substance designed to look, taste, and feel the same as the test drug. This is one way of separating the true effect of the drug from other effects, such as simply being involved in a study. Example time (skip if you already know this stuff).
Hypothesis: Oral Wonderflonium increases scrotal volume in human adult males. This is based on a mouse model that showed some promising results. Aspiring Academic Medical Center (AAMC) decides to test the hypothesis. They recruit 400 human adult males and measure their scrotal volumes on entry into the study (up to you to imagine how it’s done). They treat the males with Wonderflonium for six months and then re-measure the scrotal volumes. Lo and behold, the average scrotal volume of the group increases by 20%. The folks at AAMC submit the paper to Second Tier Medical Journal.
Two of the reviewers at STMJ love it. Increased scrotal volume! Wonderflonium! Twenty percent! The final reviewer is less enthusiastic. “How do we know it was the Wonderflonium? After all, we’re not comparing these people to anyone else. Maybe the drive two and from the center increased their scrotums. Maybe there’s something at the coffee cart. Go back and re-work this.”
The AAMC folks want to re-do the study, but ran out of money, as Wonderflonium is very hard to come by. But over at Major Academic Medical Center (MAMC), there’s a guy with a big grant who grabs the whole idea by the, um, reins and sets up a new study. This time another 400 men are recruited, but half receive Wonderflonium and half receive a placebo that is outwardly identical to Wonderflonium.
This time, the results are published: Wonderflonium increased scrotal volume by 18% over placebo. Woo. Hoo.
The potential market is huge, especially among a certain set of inadequate-feeling, ‘roid-raging gym rats. Several more studies are designed by academics and by drug companies (sometimes indistinguishably). Wonderflonium is flying off the shelves as the studies continue to give positive results.
A few years later, someone goes back and looks at the data. The most recent studies are showing that scrotums among gym rats aren’t actually getting much bigger, something expected given the wide use of the drug. He goes back and finds recent studies have shown that scrotal size is only up by about 3-4% vs. placebo.. WTF?
The researcher notes that scrotums in the placebo groups in the latest study are up more than in the original, and that this increase in placebo response is responsible for what seems like a decrease in the drug’s efficacy.
His colleague has a different idea, one examined in detail by Dr. David Gorski at Science-Based Medicine. His colleague looks at the trend, and she sees something familiar: new drugs seem to work better right after their discovery. She proposes the Scrotum Problem is due to the “decline effect”, a mixture of factors that better explains the trend. The early studies of the new drug may have captured a less random moment in time. As more and more people are treated, there is a “regression toward the mean”, with more studies giving us more accurate (and less impressive) numbers. There may also have been some publication bias, in which studies with interesting positive results were more likely to be submitted and published.
What is less likely is that something about placebos has changed. While placebo scrotum volumes increased, there may be many explanations for this. Study design can affect placebo response considerably, as can measurement biases. But since placebos don’t change, they’re not to blame.
So before we all get too excited about the increasing placebo effect killing off new drug development we need to ask smarter questions: are we looking at the right end points and doing it correctly? Are we allowing expectations to trump good study design? Are we jumping the gun on new drugs by relying on initial small studies? Are we moving the goal posts in our original studies by hunting for statistically significant effects that we didn’t lay out in our hypothesis?
All could be true. But a sugar pill is still a sugar pill.