UK United Kingdom

From human extinction to super intelligence, two futurists explain

The Conversation organised a public question-and-answer session on Reddit in which Anders Sandberg and Andrew Snyder-Beattie, researchers at the Future of Humanity Institute at Oxford University, explored…

The future is uncertain, and that’s a problem. cblue98, CC BY-SA

The Conversation organised a public question-and-answer session on Reddit in which Anders Sandberg and Andrew Snyder-Beattie, researchers at the Future of Humanity Institute at Oxford University, explored what existential risks humanity faces and how we could reduce them. Here are the highlights.

What do you think poses the greatest threat to humanity?

Sandberg: Natural risks are far smaller than human-caused risks. The typical mammalian species lasts for a few million years, which means that extinction risk is on the order of one in a million per year. Just looking at nuclear war, where we have had at least one close call in 69 years (the Cuban Missile Crisis) gives a risk of many times higher. Of course, nuclear war might not be 100% extinction causing, but even if we agree it has just 10% or 1% chance, it is still way above the natural extinction rate.

Nuclear war is still the biggest direct threat, but I expect biotechnology-related threats to increase in the near future (cheap DNA synthesis, big databases of pathogens, at least some crazies and misanthropes). Further along the line nanotechnology (not grey goo, but “smart poisons” and superfast arms races) and artificial intelligence might be really risky.

The core problem is a lot of overconfidence. When people are overconfident they make more stupid decisions, ignore countervailing evidence and set up policies that increase risk. So in a sense the greatest threat is human stupidity.

In the near future, what do you think the risk is that an influenza strain (with high infectivity and lethality) of animal origin will mutate and begin to pass from human to human (rather than only animal to human), causing a pandemic? How fast could it spread and how fast could we set up defences against it?

Snyder-Beattie: Low probability. Some models we have been discussing suggest that a flu that kills one-third of the population would occur once every 10,000 years or so.

Pathogens face the same tradeoffs any parasite does. If the disease has a high lethality, it typically kills its host too quickly to spread very far. Selection pressure for pathogens therefore creates an inverse relationship between infectivity and lethality.

This inverse relationship is the byproduct of evolution though – there’s no law of physics that prevents such a disease. That is why engineered pathogens are of particular concern.

Is climate change a danger to our lives or only our way of life?

Sandberg: Climate change is unlikely to wipe out the human species, but it can certainly make life harder for our civilisation. So it is more of a threat to our way of life than to our lives. Still, a world pressured by agricultural trouble or struggles over geoengineering is a world more likely to get in trouble from other risks.

How do you rate threat from artificial intelligent (something highlighted in the recent movie Transcendence)?

Sandberg: We think it is potentially a very nasty risk, but there is also a decent chance that artificial intelligence is a good thing. Depends on whether we can make it such that it is friendly.

Of course, friendly AI is not the ultimate solution. Even if we could prove that a certain AI design would be safe, we still need to get everybody to implement it.

Which existential risk do you think we are under-investing in and why?

Snyder-Beattie: All of them. The reason we under-invest in countering them is because reducing existential risk is an inter-generational public good. Humans are bad at accounting for the welfare of future generations.

In some cases, such as possible existential risks from artificial intelligence, the underinvestment problem is compounded by people failing to take the risks seriously at all. In other cases, like biotechnology, people confuse risk with likelihood. Extremely unlikely events are still worth studying and preventing, simply because the stakes are so high.

Which prospect frightens you more: a Riddley Walker-type scenario, where a fairly healthy human population survives, but our higher culture and technologies are lost, and will probably never be rediscovered; or where the Earth becomes uninhabitable, but a technological population, with cultural archives, survives beyond Earth?

Snyder-Beattie: Without a doubt the Riddley Walker-type scenario. Human life has value, but I’m not convinced that the value is contingent on the life standing on a particular planet.

Humans confined to Earth will go extinct relatively quickly, in cosmic terms. Successful colonisation could support many thousands of trillions of happy humans, which I would argue outweighs the mere billions living on Earth.

What do you suspect will happen when we get to the stage where biotechnology becomes more augmentative than therapeutic in nature?

Sandberg: There is a classic argument among bioethicists about whether it is a good thing to “accept the given” or try to change things. There are cases where it is psychologically and practically good to accept who one is or a not very nice situation and move on… and other cases where it is a mistake. After all, sickness and ignorance are natural but rarely seen as something we ought to just accept – but we might have to learn to accept that there are things medicine and science cannot fix. Knowing the difference is of course the key problem, and people might legitimately disagree.

Augmentation that really could cause big cultural divides is augmentation that affects how we communicate. Making people smarter, live longer or see ultraviolet light doesn’t affect who they interact with much, but something that allows them to interact with new communities.

The transition between human and transhuman will generally look seamless, because most people want to look and function “normally”. So except for enhancements that are intended to show off, most will be low key. Which does not mean they are not changing things radically down the line, but most new technologies spread far more smoothly than we tend to think. We only notice the ones that pop up quickly or annoy us.

What gives you the most hope for humanity?

Sandberg: The overall wealth of humanity (measured in suitable units; lots of tricky economic archeology here) has grown exponentially over the past ~3000 years - despite the fall of the Roman empire, the Black Death and World War II. Just because we also mess things up doesn’t mean we lack ability to solve really tricky and nasty problems again and again.

Snyder-Beattie: Imagination. We’re able to use symbols and language to create and envision things that our ancestors would have never dreamed possible.

Join the conversation

6 Comments sorted by

  1. Damian Macrae

    logged in via email

    Why is it that futurists always handwave sources of energy for humanity? Colour me skeptical, but I don't see much space exploration or advanced AI happening if EROEI gets into the low single digits.

  2. John Doyle


    Very interesting discussion. It links well with other future discussions doing the rounds and directly links with some of them.
    The mentioned 3000 years of increasing wealth is rapidly coming to an end. We are trying to live exponentially in a finite world and the reckoning is now upon us. From about now wealth will decline, possibly precipitously to a point the planet can begin to recover from the depredations we are visiting upon it.
    The mentioned inability to consider the welfare of future generations, the constant overconfident stupidity are going to make human recovery very difficult. Maybe it will end up as a Riddley-Walker case [?]
    This thread also raises problems of health and disease. These will of course become more prevalent during a decline when medical services are cut because they lose the infrastructural support network. The loss of technological capacity due to scarce energy will have profound negative consequences.

    1. Damian Macrae

      logged in via email

      In reply to John Doyle

      Yes, and lets remember that the '3000 years of exponential improvement' is basically a lie. The average Roman and average Tudor lived essentially the same life (Romans probably better if we are honest) -> it was only after coal and liquid fossil fuels that things really took off...

  3. Glenn Baker


    The more technology advances the more dependant we become and we can already see dangerous times ahead. The Riddley Walker is most likely one and in my opinion the best. Only the healthy will adapt and survive and dangerous technologies will be gone at least for a few centuries. Remember the industrial revolution started the rot; a huge population explosion; global warming and enormous energy requirements per capita.


    logged in via Twitter

    Futurism - shmuturism. The future is already here: climate change and fossil fuel dependency are existential risks so large they stare us into the face, while we reach for the stack of old futurist vignettes for pure distraction and denial.