Experts such as epidemiologists and statisticians made much more accurate predictions about COVID than the public, but both groups substantially underestimated the true extent of the pandemic, a study from the University of Cambridge has found.
Researchers from the Winton Centre for Risk and Evidence Communication surveyed 140 UK experts and 2086 UK laypersons in April 2020 and asked them to make predictions about the impact of COVID by the end of 2020. Participants were also asked to assign confidence in their predictions by providing upper and lower bounds of where they were 75% sure that the true answer would fall—for example, a participant would say they were 75% sure that the total number of infections would be between 300 000 and 800 000.
While only 44% of predictions from the expert group fell within their own 75% confidence ranges, only 12% of predictions from the non-experts fell within their ranges, though more numerate individuals performed a little better. The results were published in the journal PLOS ONE.
“Experts perhaps didn’t predict as accurately as we hoped they might, but the fact that they were far more accurate than the non-expert group reminds us that they have expertise that’s worth listening to,” said lead author Dr Gabriel Recchia from the Winton Centre for Risk and Evidence Communication,. “Predicting the course of a brand-new disease like COVID-19 just a few months after it had first been identified is incredibly difficult, but the important thing is for experts to be able to acknowledge uncertainty and adapt their predictions as more data become available.”
Expert opinion is important for those making decisions at any level from individual to policy. The quality of expert intuition can vary greatly depending on the field of expertise and the type of judgment required, so it is important to determine how good expert predictions really are, especially in where they could shape public opinion or government policy.
“People mean different things by ‘expert’: these are not necessarily people working on COVID-19 or developing the models to inform the response,” said Dr Recchia. “Many of the people approached to provide comment or make predictions have relevant expertise, but not necessarily the most relevant.” Dr Recchia noted that in the early stages of the pandemic, clinicians, epidemiologists, statisticians, and other individuals seen as experts by the media and the general public, were often asked to give off-the-cuff answers to questions about how bad the pandemic might get. “We wanted to test how accurate some of these predictions from people with this kind of expertise were, and importantly, see how they compared to the public.”
Participants in the survey were asked to predict how many people living in their country would have died and would have been infected by the end of 2020; they were also asked to predict infection fatality rates both for their country and worldwide.
The expert group and the non-expert group both underestimated the total number of deaths and infections in the UK. The official UK death toll at 31 December was 75 346. The median prediction of the expert group was 30 000, while that of the the non-expert group was 25 000.
For COVID fatality rates, the median expert prediction was that 10 out of every 1000 people with the virus worldwide would die from it, and 9.5 out of 1000 people with the virus in the UK would die from it. The median non-expert response to the same questions was 50 out of 1000 and 40 out of 1000. The true infection fatality rate at the end of 2020—as best could be estimated—was nearer to 4.55 out of 1000 worldwide and 11.8 out of 1000 in the UK.
“There’s a temptation to look at any results that says experts are less accurate than we might hope and say we shouldn’t listen to them, but the fact that non-experts did so much worse shows that it remains important to listen to experts, as long as we keep in mind that what happens in the real world can surprise you,” said Dr Recchia.
The researchers cautioned that it is important to differentiate between research on evaluating the forecasts of ‘experts’—individuals involved in relevant fields, such as epidemiologists and statisticians—and research on evaluating specific epidemiological models, though the models may inform experts. Many COVID prediction models have proved accurate in the short term, but rapidly become less accurate for later predictions.
Source: Medical Xpress
Journal information: PLOS ONE (2021). DOI: 10.1371/journal.pone.0250935
