Scientists feud over hyped Stanford coronavirus antibody study: 'How NOT to do statistics'

Antibody testing is necessary to determine the prevalence of the coronavirus in the population and reopen the economy. Yet a new study suggesting that Santa Clara County, home to Silicon Valley, had up to 85 times as many cases as it reported was deeply flawed and could give the public a false sense of security, researchers say.

Unlike diagnostic tests, which show when a person is infected, serological tests detect antibodies that your body creates as it fights the virus, and even after it clears itself of it. Experts believe that many more people have contracted the coronavirus and recovered than has officially been reported, especially since 25% or more cases may be asymptomatic. Many companies and universities are rolling out their own tests to confirm these trends. Yet experts have raised questions about how accurate these tests are, and the statistical assumptions that researchers infer from their results.

Researchers at Stanford University released a study last week suggesting that based on 3,300 antibody tests that up to 4.16% of the population in California's Santa Clara County — about 81,000 people — had been infected. The study, which like many released amid the pandemic, was first published on the preprint server medRxiv and has not yet been peer-reviewed, found that the total number of cases in the county could be up to 85 times higher than the reported 956 cases.

Some of the researchers involved also participated in a University of Southern California (USC) study that similarly estimated based on 864 tests that up to 5.6% of adults in Los Angeles County had been infected, suggesting a much higher total than the official count of 7,994. That study has not been released, though the findings were published in a press release.

The findings drew praise from protesters calling for an end to stay-at-home orders who believe that the virus is far less deadly than first thought, The Guardian reported. Others accused the authors of using the study to push a pre-existing agenda. Two of the researchers involved in both studies, Stanford's Eran Bendavid and Jay Bhattacharya, published an editorial in the Wall Street Journal in March arguing that stay-at-home orders were not warranted because the virus was not as deadly as first thought. Another Stanford researcher involved in the study, John Ioannidis, also published an editorial arguing that lockdown decisions were made without reliable data.

Many researchers around the country said the Santa Clara County study was simply flawed. The researchers used kits from the company Premier Biotech, which has one of the lower "false positive" rates but still shows "false positives" in two of every 371 true negative samples, according to the San Jose Mercury News.

"With that ratio of false positives, a large number of the positive cases reported in the study — 50 out of 3330 tests — could be false positives, critics note," the outlet reported. "To ensure a test is sensitive enough to pick up only true [coronavirus] infections, it needs to evaluate hundreds of positive cases of COVID-19 among thousands of negative ones."

The study was also criticized for recruiting volunteer test subjects on Facebook, which critics argued could have skewed the testing sample to people who believed they had been infected but were unable to get tested.

"I think the authors of the above-linked paper owe us all an apology," said Andrew Gelman, the director of the applied statistics center at Columbia University. "We wasted time and effort discussing this paper whose main selling point was some numbers that were essentially the product of a statistical error."

The study's authors told the Mercury News that they would release a detailed appendix that would address many of the "constructive comments and suggestions" that the researchers had received.

"This is exactly the way peer-review should work in scientific work, and we are looking forward to engaging with other scholars as we proceed in this important work," Bhattacharya told the outlet.

Ioannidis also said that an expanded version of the study would soon be posted and argued that "the results remain very robust."

But critics said the results, which have already been widely shared, were an example professors could teach to classes to "show how NOT to do statistics."

"They're the kind of screw-ups that happen if you want to leap out with an exciting finding," said Gelman, "and you don't look too carefully at what you might have done wrong."

"Do NOT interpret this study as an accurate estimate of the fraction of population exposed," warned Marm Kilpatrick, an infectious disease researcher at the University of California Santa Cruz. "Authors have made no efforts to deal with clearly known biases and whole study design is problematic."

The USC study, which used the same type of test, was criticized for broadcasting its results without describing its methodology. Los Angeles officials have already pegged their death rate at 0.2% based on the results, however, suggesting that the virus is twice as deadly as the flu but far less deadly than other estimates.

Critics cast doubt on whether researchers could fix the problem with false positives in the studies.

"In a low risk population, false positives quickly outnumber true positives that the results are very difficult to interpret. Even with a positive test, you're more likely than not to be negative," said Natalie Dean, a biostatistician at the University of Florida.

Dr. Deborah Birx, the coordinator of the White House coronavirus task force, explained the difficulty in drawing conclusions from results with even very small "false positive" rates during a news briefing on Monday.

"So if you have 1 percent of your population infected and you have a test that's only 99 percent specific, that means that when you find a positive, 50 percent of the time will be a real positive and 50 percent of the time it won't be," she said.

Caitlin Rivers, an epidemiologist at the Johns Hopkins Center for Health Security, argued that antibody tests will certainly show that the coronavirus death rate is considerably lower than the total deaths compared to the total confirmed cases "but it's excruciatingly clear that this virus is many times worse than influenza and capable of overwhelming health systems."

"Doing careful science is tricky even in the best of times… usually takes teams of experts and peer reviewers to get a good piece of work over the finish line. In a crisis, the normal process is rushed or skipped and it can hurt quality," she explained. "New findings should be added to the weight of evidence. Science is an accumulation of knowledge. Rarely are there blockbuster findings that completely change our understanding. So next time you see a claim that seems too clever, it probably is."

The World Health Organization (WHO) cautioned that it would be premature to make decisions regarding lockdowns and social distancing requirements because only a small percentage of the population is believed to have developed antibodies.

"Early data suggests that a relatively small percentage of the populations may have been infected," WHO Director-General Dr. Tedros Adhanom Ghebreyesus said on Monday. "Not more than 2%-3%."

Dr. Maria Van Kerkhove, who is leading the organization's response to the pandemic, added that "we see a lower proportion of people with antibodies than we were expecting."

New York, the hardest-hit area in the world, released preliminary results of its antibody testing, estimating based on 3,000 tests. Those results predicted that up to 21% of New York City's population and more than 13% of the statewide population had been infected and recovered. Though the numbers are much higher than the ones in the California study, the researchers estimated that about 2.7 million people in the state of New York had been infected, only about 10 times more than the number of confirmed cases.

But hours before New York Gov. Andrew Cuomo announced the results, Dr. Demetre Daskalakis, the top disease control official in New York City, warned that the antibody tests "may produce false negative or false positive results" and therefore "should not" be used to determine whether those that test positive for antibodies have immunity to the virus.

Daskalakis noted that a negative result would not exclude an infection "in a patient with recent exposure to the virus" because antibodies may take days to become detectable.

"Cross-reactivity of antibody to other common coronavirus proteins may also occur, so a positive serology may either reflect infection with SARS-CoV-2 or past infection with other human coronaviruses," he added.

Researchers said it was too early to determine what it actually means to have antibodies to the virus.

"I'm very ambivalent about these tests, because we don't really know yet through the science what it means to have an antibody," Dr. Joan Cangiarella, a pathologist at New York University's Langone Health, told The New York Times. "We are hoping these antibodies mean you will be immune for some time. But I don't think the data is fully out there to understand if that means that you're actually immune. And if these antibodies start to decline, what's that time frame? Does it decline in a year from now?"

Dr. James Crawford, who oversees the laboratory network at New York's Northwell Health system, added that there was a "pressing need to have both parts of the equation — who has the virus and who has mounted an immune response."

"To use it as a fulcrum for when someone can travel or work," he cautioned, "I think we have to be extremely careful as a society in doing that."