By Erik Sass, TES Editor-in-Chief
As controversy continues to rage about the efficacy of hydroxychloroquine and other drugs against COVID-19, regulators and scientific authorities in the U.S. and abroad have consistently held up randomized controlled trials (RCTs) as the “gold standard” of drug efficacy research – but what if there was actually a way to “hack” these supposedly bias-free clinical experiments to show any result an interested party might wish, with scientists none the wiser? Taking a closer look at the innovative (and largely untested) online recruitment methods behind some of the most-cited RCTs on COVID-19 treatments, it appears that this speculation is all too plausible. Medical researchers, regulators, and the general public should be aware of this real threat, especially in light of the clear trend to Internet-based recruitment for RCTs.
This year has already seen a number of examples of RCTs specifically considering hydroxychloroquine as a treatment for COVID-19 using remote online recruitment of subjects, including “A Randomized Trial of Hydroxychloroquine as Postexposure Prophylaxis for Covid-19” (COVID-19 PEP) and the COVID-19 Post-Exposure Prophylaxis (PEP Study). The most useful examples for this article are the heavily cited RCTs (1, 2, 3) from researchers at the University of Minnesota, where the studies’ lead authors and organizers have provided additional insights into methodology in media interviews.
In early July 2020 Dr. David Boulware, who had a lead role organizing the UM RCTs, provided a description of his team’s efforts during a podcast interview, which was published on 7 July, 2020 here. This interview provides details of how the work for all three of the UM RCTs was carried out.
In one exchange the interviewer noted the possibility that, because online recruitment was based largely on self-reporting of symptoms, with medications delivered by mail or package service, some self-interested individuals (e.g., hoarders) might enroll simply to gain access to HCQ whether they had symptoms or not. However Boulware expressed confidence that because the study was “blinded” to participants – meaning subjects didn’t know whether they were getting HCQ or just a placebo – its design would effectively deter freeloaders:
That is the beauty of the randomized trial. For most people I think they were pretty serious. If people just wanted hydroxychloroquine you could just write a prescription or have someone write a prescription for you and take it… Yeah, you are twice as likely to get the real drug that way. So if you were to randomize and say, “Man, am I going to get a vitamin or I’m going to get hydroxychloroquine?” If people really wanted hydroxychloroquine they were probably not going to enroll in our trial. But that is the beauty of a randomized trial in that if we had fake people or people that got lost to follow-up because they didn’t really want to be in the trial, they got equally balanced between the two arms.
However the same study conditions point to a potentially fatal flaw. While the study design may or may not have served to deter hoarders or other self-interested individuals, this does not account for the possibility of intervention by a group or organization determined to portray HCQ as ineffective (other shortcomings of these RCTs are discussed in The Economic Standard’s paper on HCQ).
With six decades of use, hydroxychloroquine is a well-known and well-characterized drug. This means an enrollee with some technical knowledge could perform an analysis of the pill received in the mail to determine whether HCQ was present, using methods that are “simple, sensitive,” and relatively easily accessible. With this in mind a group of “sophisticated enrollees” would only have to take the following steps to effectively “hijack” the RCT:
1. An online application would need to be submitted to join the study. The “fake” enrollee would not need to be sick. At no point was an enrollee required to see a doctor.
2. Upon receiving the drug of unknown identity in the mail, a sophisticated group could test it to determine whether it contained HCQ or placebo. As detailed in the link above, such an entity could perform such tests with relative ease using a spectrophotometer, an instrument widely used in the health profession and educational environment. Once the traits of the drug were identified for one purported enrollee, more rudimentary means (e.g., appearance) might be used to allow other individuals to determine whether a drug was HCQ or placebo.
3. If the drug were found to be the placebo, the (fake) enrollees would submit symptom data which reflected rapid recovery. If the drug contained HCQ, they would report slow resolution of symptoms. Of note, at no point did any participant need to communicate with, or be seen by, a doctor, nor did they have to actually take the medication.
Based on the lead researcher’s own statement, one might ascertain that there was a belief that the prospect of manipulation via the enrollees was believed to be insignificant. But the process outlined above shows that a motivated entity, who wished to show no benefit for HCQ against COVID-19, could affect the symptom severity score for this particular RCT with relatively little effort.
In other words, a motivated and sophisticated entity could have manipulated data in both arms to show HCQ to be ineffective and the placebo to be highly effective. An essential point is that such a manipulation might have occurred without the study authors’ knowledge.
It would only take a relatively few patients to affect the symptom score data, including enlarging the error bars which were primarily responsible for the authors’ conclusions that no effect was seen.
The possibility of ‘bad actors’ among study enrollees must be considered by researchers and scientists, especially as new remote study procedures are utilized for RCTs aimed addressing the COVID-19 pandemic. While it might seem beyond our comprehension that any “motivated entity” would “hijack” a scientific study with life-or-death implications for thousands of people, the retracted articles in the Lancet and NEJM suggest that there may be more at work than meets the eye.