Governments around the world, driven by the threat of overburdened health systems and mass mortality resulting from the COVID-19 pandemic, are being forced to make decisions that have enormous, long-lasting consequences for lives and economies. They are doing so without sufficient assurance that the choices they make are the best ones.
The fog of uncertainty can be partly lifted by better use of information that’s emerging around the world. But it will never completely clear.
The most difficult choices are those that have to be made before you know how they will work out.
The challenge of making high-consequence decisions based on imperfect knowledge is not unprecedented. For example, tough policies to mitigate climate change must be adopted long before the world crosses catastrophic thresholds. To guide these decisions, countries rely on imperfect models of the climate system, along with divergent and values-based assumptions about how the future could unfold.
Scientists and policymakers, working together over a period of three decades through the Intergovernmental Panel on Climate Change, have worked out how to guide decisions in the face of uncertainty in such a way that there is broad agreement, and which minimises regrets even if the future does not work out exactly as projected. Their approach has allowed scientists to remain providers of evidence, and politicians to focus on value-based choices.
The Paris Agreement to mitigate climate change, signed by 197 countries, was the result of a well-functioning science-policy interaction.
This experience can be applied to the response to the COVID-19 pandemic, without having to reinvent the process.
The best practice guidelines involve making decisions based on the best available information at the time, and progressively improving them in the light of experience and emerging new information. Secondly, they involve using a “multi-model approach” and an ensemble of results, rather than placing all bets on a single prediction. And finally, they use collectively agreed scenarios to explore the full range of options and outcomes.
Decisions about containing COVID-19 are inevitably a balancing act between reducing the immediate loss of lives on the one hand, and protecting livelihoods that could be damaged as a result of the actions taken on the other. The models used to support the decisions must be similarly balanced. There is no point in having precise projections about the course of the pandemic, but only a vague idea of the impact on the economy.
At present, these different streams of information are not well integrated.
For the case of using mathematical models to help guide COVID-19 policies, we make the following suggestions based on our collective experience with scientific assessments.
Use what information is available, then adapt: The novelty of the disease means you start from knowing very little and taking guidance from experiences with similar diseases in the past. You work towards improving modelled projections, using information from a range of sources – from science to public health to the economy.
Nimble and efficient channels of communication ensure that the pace of modelling matches the urgency of the problem.
The multi-model approach: Using several different models rather than one relies on the same logic that tells you not to put all your savings into a single asset. The most robust approach is to build a portfolio, which is collectively stronger than just one, particularly if they are based on fundamentally different assumptions.
Typically, different models have different purposes, and some are stronger in some respects than others. Some models are good at short-term projections while others are better in the long term. Including more detail is necessary for some purposes, but a less detailed model may be sufficient, and more reliable, for more general policies.
This does not mean you should not winnow out models that are simply wrong. But to do so you need a well-structured, evidence-based test. The statistician George Box wisely commented that “all models are wrong, but some are useful”.
For modelling COVID-19 we would similarly encourage a diversity of models.
Scenarios: Some things cannot be predicted accurately, because they depend on chaotic physical processes, or behaviours that defy simple representation, such as human choices. For these issues you use scenarios. Scenarios allow the models to be stress-tested, by asking questions such as: What is the range of possible outcomes? How does my decision play out in the worst case, as well as in my preferred case?
The scenarios must be shared between models, or you are unable to tease apart differences in the way models work from differences in model drivers.
The scenarios need to be plausible, but must span a wide range of possibilities if they are not to lead to confirmation bias – our tendency to choose the outcomes that support our prejudices. It is important to include measurable indicators, so that you later know which scenario is playing out.
For COVID-19 we recommend exploring the model predictions over a range of agreed scenarios. For example, one scenario can impose strict lockdown and maintain it over several months. Another can progressively relax the restrictions. And both can be compared to a reference scenario where no policy action is taken.
When many models, several scenarios and uncertain data are used together, the result will be a wide range of predictions. The differences need to be evaluated so it’s clearer which findings have the most supporting evidence.
Public trust is key
The balancing act of managing COVID-19 requires public trust, which is fostered by an open, clear and credible process of decision-making. The framework we propose is focused on providing the information needed to make good decisions, but should not assume the right to make the decisions. For that purpose, people elect political leaders to represent their rights and values.
This approach has been successfully applied elsewhere, for instance in the protection of the ozone layer, and mobilising action to halt biodiversity loss. In South Africa, it recently aided sensible decisions regarding fracking in the Karoo.