Risks and dire scenarios. Why AI amplifies our fears

There is always a need for an imminent end of the world. This week, two news stories made the rounds in the media landscape. On 9 September, Jacob Coxon, a researcher who had worked for both OpenAI and Anthropic, announced his resignation, claiming that the two companies were racing towards an artificial intelligence so powerful that it would put our lives at risk. The following day, Anthropic published a new report on the misuse of its models, also documenting attempts to use Claude for research potentially relevant to the development of biological weapons. Anthropic’s report was interpreted as confirmation of Coxon’s statements, and both have reinforced a narrative the media has been flirting with for months: artificial intelligence as a weapon aimed at humanity. This is not a matter of sceptical views on AI’s actual potential for growth, as found amongst economists, nor about issues relating to the impact of language models on this or that profession. The threat is said to be collective, universal and unstoppable.

Michael Shermer has noted that turning the end of the world into a ‘scenario’ guarantees significant media attention and comes at very little cost to those who predict it: media interest fades just as quickly as it flared up, and no one takes offence if the prophecy fails to materialise. We could leave it at that, emphasising how the past is full of predicted ends of the world and devoid of any that have actually happened. The argument, however, has a limitation: if one of those predictions had come true, there would be no one left to record it. Let’s try to take these arguments seriously. Coxon maintains that those building AI believe it is capable of killing us all by the end of the decade; Evan Hubinger, who leads Anthropic’s Alignment Stress-Testing group, assigns a probability of over 10 per cent to this possibility (again, within the next ten years). These assessments relate to future scenarios involving a possible self-improving superintelligence, not current models. To use Frank Knight’s terminology, we are in the realm of uncertainty. Knight spoke of risk when the probability of outcomes is measurable and of uncertainty when such a measurement is not possible.

This 10 per cent is not a probability derived from observed frequencies or from an empirically validated model: it is an expert’s view, a scenario built on a chain of highly uncertain causal hypotheses. Will there be a similar disruption? What forms might it take? The uncertainty, however, works both ways: if the probability of a catastrophe cannot be estimated, one cannot assign it a high value, nor even a low one. The problem becomes how to decide when the figures are lacking.

The history of artificial intelligence is replete with examples of authoritative predictions that placed capabilities in the near future which have turned out to be much further off than anticipated. Before the large-scale development of large language models (LLMs), many believed this line of research was a dead end. The current acceleration is real and unprecedented. However, this does not mean that AI development will necessarily be ‘exponential’ from now on. Recursively self-improving superintelligence remains a working hypothesis.

The case of the report on biological risks is different. Here we are dealing with observed cases, albeit few in number, and Anthropic bases its assessment on its own data and certainly knows how to interpret it. Over the nine-month period in question, some users had employed Claude for research into dual-use biology – that is, research that could be used both for therapeutic purposes and, potentially, for the manipulation of pathogens. In the most widely discussed case, relating to a proposed gain-of-function research project on chikungunya, Anthropic states that it was unable to establish malicious intent. It blocked the request nonetheless, applying a very strict precautionary principle: when faced with a specific and serious danger, doubt regarding intent is sufficient grounds to intervene. In essence, the episode merely confirms that there are individuals who misuse AI for harmful purposes. The company’s intervention is an example of self-regulation: Anthropic established internal rules and applied them rigorously. It has strong incentives to act in this way; if one of the scenarios feared by doomsayers were to actually come to pass, the repercussions could be severe. Recent analyses of terrorism point in a similar direction. AI can increase the efficiency, accessibility and scale of certain criminal activities, particularly phishing, deepfakes and cyber-attacks. There remains limited evidence that it has substantially altered the operational capabilities required to carry out physical attacks. A language model does not remove the main obstacles to carrying out such an attack: access to operatives or explosives, tacit knowledge, organization, secrecy and access to the target. Demonstrating that AI can provide technical assistance is not the same as demonstrating that it has significantly increased the actual risk.

In public discourse, however, Ragnarök looms on the horizon. Why? Essentially, for the same reason we are more interested in bad news than good. The human mind is probably better equipped to detect threats than to accurately assess their likelihood. Our cognitive systems are prone to false positives. They evolved in environments characterised by widespread threats and frequent episodes of violence. In that context, mistaking a harmless sound for a predator comes at little cost; the opposite mistake can be fatal. AI amplifies our fears for at least three reasons. Firstly, it affects language, a central capacity in our species’ self-representation: the threat is therefore also one of identity. On the one hand, we would like to dismiss the machine as a ‘stochastic parrot’; on the other, we imagine it will lead us to extinction. The same individuals manage to hold two such blatantly contradictory beliefs. Secondly, for the first time, a mass-market technology converses with its users. The ELIZA effect demonstrated as far back as the 1960s just how easily natural language leads us to attribute intentionality and agency. Systems capable of adapting to and personalising the interaction make this projection even more powerful. Thirdly, many of the professional groups that perceive the threat most directly – journalists, translators, academics – are precisely those who produce a large part of public discourse. This introduces a selection bias into the social representation of technology.

Our ancestors heard the wind whistling through the bushes and had two options. They could reason about the likelihood that the bush concealed a ferocious beast. Or they could flee. Those who fled were more likely to survive. We are their descendants, and thinking in terms of probability does not come naturally to us at all. In public debate, this reflex takes the form of the precautionary principle: a technology must be halted until its safety is proven. Cass Sunstein has observed that, taken literally, the principle contradicts itself, because even abandoning a technology entails risks that are just as difficult to assess: such as the very uses of AI that might result in biological weapons, undiscovered therapies and slowed-down research. Aaron Wildavsky, in Searching for Safety, argued that societies become safer primarily through trial and error, accumulating knowledge and resources with which to respond to dangers as they arise. Proactive prevention works when the danger is known and contained, as in the case of chikungunya. Extending it to an entire technology, in the name of scenarios with vague outlines, deprives us of the knowledge that accumulates through use.