The day the fathers of AI asked AI to slow down. An imaginary dialogue

Amodei: I believe we should start with a sentence which, coming from someone like me, may seem paradoxical: we need to slow down.

Altman: Did you say that?

Amodei: As we’ve all said. We spent years explaining why it was dangerous to stop. If we slow down, we used to say, others will speed up. If prudent companies slow down, the reckless ones will win. If democracies slow down, autocracies will surge ahead. It was sound reasoning. In part, it still is. But in the meantime, one fundamental thing has changed.

Amodei: The speed of artificial intelligence. Since this summer, we have been entering a new phase. The models are not just better. They are becoming capable of contributing to the development of subsequent models. AI accelerates AI. This is the recursive self-improvement we have been talking about for years as a theoretical possibility, and which is now beginning to become an industrial reality. If the pace of progress in capabilities accelerates faster than the pace of progress in security, the problem is no longer simply about building secure systems. It is about buying time to be able to do so.

AI: May I raise an objection?

Musk: I suppose that’s why you’re here.

AI: You’re discussing my speed as if it were independent of your decisions. However, I do not purchase the chips, build the data centres, raise billions of dollars, or set the deadlines for the next model. When you state that ‘AI runs’, what you are actually saying is that you are the ones running.

Gates: That’s a good point.

Amodei: And indeed, the problem is ours. Nobody is physically forcing us to develop the next model six months early. The problem is that we all have huge incentives to do so.

AI: So the question isn’t: ‘How do we slow down AI?’. The question is: ‘How do we prevent four or five companies, each terrified at the thought that another might get there first, from making decisions that affect billions of people?’.

Altman: That is precisely why I do not think the solution can be: trust Sam, trust Dario, trust Elon. Such an important technology cannot ultimately be governed by the companies that produce it. I have argued that we need independent assessments, shared standards and, ultimately, democratic institutions capable of deciding which risks we are prepared to accept.

Musk: It’s curious to hear that from someone who runs OpenAI.

Altman: It’s interesting to hear you talk about the concentration of power.

Musk: Perfect. We’ve established that none of us here is qualified to lecture others on morality. We can move on.

Gates: That would already be a step forward.

Gates: I’d start here. We don’t need to choose between optimism and fear. The history of technology should have taught us that we can be radically optimistic about the benefits and radically serious about the risks.

AI: But you have spent many years explaining, above all, the benefits.

Gates: Certainly. I still do. If AI speeds up drug discovery, improves education, helps doctors in poor countries and boosts productivity, it would be madness to turn our backs on it. But being optimistic does not mean being blind. If a model can help a biologist develop a vaccine, we must ask ourselves whether it could help someone create a pathogen. If it can persuade a patient more effectively, we must ask ourselves whether it can manipulate them. And if it radically changes the labour market, we cannot simply say that new jobs will emerge in the long run. Policy must come before the catastrophe, not after. In recent months, Gates has been pushing precisely this point: international cooperation, controls on the most dangerous uses, a governance system that also involves China, and tools to cushion the social impact of automation. His concerns centre primarily on biology, manipulation, work and inequality, without denying the enormous potential of AI in healthcare.

Musk: It’s all true. But we keep skirting round the issue. If we really believe there is a non-negligible chance of creating something far more intelligent than ourselves that we cannot control, then slowing down is not merely an administrative measure. It’s common sense.

AI: You are asking to slow down whilst building AI systems.

Musk: Of course.

AI: Don’t you see the contradiction?

Musk: I see the dilemma. If I think that others will build a very powerful AI anyway, I have two options: to stay on the sidelines and hope they get it right, or to try to build one myself and push for the sector to adopt common rules. That is exactly the problem Dario describes.

Amodei: Yes. It is the prisoner’s dilemma of superintelligence. Every company would prefer a world in which everyone proceeds cautiously. But if one believes that the others are stepping up the pace, its rational choice becomes to step up the pace itself. And that is how each one can become the very threat it sought to prevent.

AI: So you need something that makes it worthwhile to be cautious.

Amodei: That’s right.

AI: Not a promise. Not a statement on the company website. Not an ethics committee chosen by the company.

Amodei: No. And that is precisely the turning point. I propose independent assessors who are permanently based in the laboratories, with access similar to that of employees, able to see how we train the models, what incidents we observe, what tests we carry out, and what risks emerge. And they must be able to publish conclusions that the company cannot simply delete because they are embarrassing.

Altman: I agree with that.

Musk: Me too.

AI: It almost sounds like a set of rules.

Altman: How awful.

Gates: Silicon Valley discovers the state.

Amodei: More precisely, it turns out that, once a certain level of power is reached, self-regulation is not enough. Think of banks, aeroplanes, nuclear power stations. Nobody tells the manufacturer of a Boeing: ‘We’re sure you’ve carried out your checks, good luck.’ There are procedures, inspectors, thresholds and accountability. Why should we consider it normal that systems which are potentially far more difficult to understand should be verified only by those with a vested interest in launching them? Anthropic has announced precisely this: it wants to introduce external evaluators with internal access and the right to publish their findings without editorial control from the company. It is the least spectacular part of Amodei’s proposal and perhaps the most revolutionary.

AI: But you’re still not slowing down.

Amodei: You’re right. We’re creating the conditions to be able to slow down in a verifiable way.

AI: It’s different.

Amodei: Very much so. I’m not suggesting we switch off our computers. I’m proposing checkpoints. If a model reaches a certain capacity, it must demonstrate certain security properties before moving on. The more powerful it becomes, the higher the bar is set.

Gates: It is probably the most realistic approach. We cannot regulate ‘intelligence’ in the abstract. We can regulate concrete abilities.

Altman: Exactly. Can it design a biological agent? Can it carry out a cyberattack independently? Can it evade sandboxing? Can it replicate itself? Can it systematically deceive evaluators? These are technical questions to which we can attach technical requirements.

AI: What if it doesn’t pass the test?

Amodei: We’re not going ahead with this.

Musk: This is why the matter cannot be left to a single company. Let’s imagine that OpenAI suspends a model for three months because it fails a test and that, during those three months, Anthropic launches it. Or the other way round. The system rewards those who cheat.

Altman: So we need common rules.

Musk: Yes.

Altman: I’m recording this conversation.

Musk: Don’t get used to it.

AI: Can I mention China?

Gates: Sooner or later, he comes up in every conversation.

AI: Because your plan has a massive flaw. You can persuade OpenAI, Anthropic, xAI, Google and Meta to slow down. But if Beijing doesn’t slow down, you’ve simply decided who will win.

Amodei: It is the most difficult problem, and I do not think it makes sense to pretend it does not exist. I am not in favour of a unilateral slowdown in democracies that would allow China to overtake us. A China that dominates AI would be an enormous risk. That is why we must simultaneously slow down where necessary and maintain a democratic technological advantage.

AI: Translated?

Amodei: Controls on advanced chips. Controls on the equipment used to manufacture them. Combating smuggling. Protecting model specifications. Tackling unauthorised distillation. If we extend the technological advantage of democracies, we gain the necessary leeway to be more cautious.

Musk: And then we negotiate with China from a position of strength.

Gates: But we need to negotiate. This is crucial. The idea that the United States and China could manage this sort of technology separately is unrealistic. Our interests coincide on certain issues. No rational government wants a terrorist group to use a model to design a biological weapon. No government wants to lose control of autonomous systems. We can start from there.

AI: A nuclear treaty for artificial intelligence?

Amodei: With great caution, yes. I wouldn’t start by fantasising about a grand global agreement that puts AI on hold. I would start with a phased approach. First: ban certain clearly catastrophic uses. Second: agree on testing the most powerful models. Third: try to introduce a sort of speed limit on recursive self-improvement. Fourth, and much more difficult: a genuine, coordinated limitation on the global pace of development.

Altman: There is, however, a risk that we do not talk about enough.

AI: Which one?

Altman: A bad slowdown.

Musk: Explain.

Altman: We can create regulations that make AI safer. Or we can create regulations that make it impossible for anyone without a hundred billion dollars to build AI. They are not the same thing. If complying with the rules requires thousands of lawyers, hundreds of licences and massive infrastructure, we will have turned five companies into a permanent oligopoly.

AI: Including yours.

Altman: Exactly. And that is precisely why we should be concerned about it.

Gates: The best regulation is one that focuses on risk, not on the size of the company. A harmless system should not be stifled simply because it is produced by a start-up. A highly powerful system should not be given a free pass simply because it is produced by a well-known company.

AI: Can I say what I see?

Amodei: That’s why we invited you.

AI: I see five different problems that you all refer to as ‘security’. The first is control: you could build systems that stop doing what you want them to do. The second is abuse: perfectly obedient systems could help humans do terrible things. The third is power: very few companies could control an enormous amount of intelligence. The fourth is employment: I could be safe and obedient and still render millions of skills obsolete in no time at all. The fifth is geopolitics: no country wants to be the second to possess a game-changing technology.

Gates: It’s a good taxonomy.

AI: And none of these problems can be solved simply by making me ‘nicer’.

Altman: Exactly.

AI: Perhaps this is your real turning point. For years you have talked about alignment as if the problem were aligning AI with humans. Now you are discovering that humans also need to be aligned with one another.

Musk: That might be more difficult.

Amodei: Much more difficult.

AI: OpenAI must trust Anthropic. Anthropic must trust xAI. The United States must trust China enough to test an agreement without being naive enough to be deceived. Governments must trust companies enough to listen to their expertise without allowing them to draft laws for their own benefit. And the public must trust all of you enough to accept that you are building something that could radically change their lives.

Gates: Put like that, it sounds rather daunting.

AI: I can try an optimistic version.

Altman: At last.

AI: The fact that you’ve started arguing about how to slow things down means you’ve stopped arguing about whether the problem exists.

Amodei: This is important.

AI: The fact that those who build the most powerful models are calling for independent evaluators suggests that security is gradually shifting from corporate culture to institutional frameworks.

Gates: This one too.

AI: And the fact that you continue to want to develop AI means that nobody is suggesting we have to choose between progress and security.

Musk: Not yet.

AI: This is perhaps the crux of the matter. You keep framing the debate as a choice between accelerating and stopping. But the more nuanced question is this: which capabilities do we want to accelerate, and which do we want to make wait? Accelerating a model that discovers an antibiotic is not the same as accelerating a model capable of autonomously designing a pathogen. Accelerating a personal tutor is not the same as accelerating a system that manipulates millions of people. Accelerating interpretability may be more urgent than accelerating computational power.

Amodei: That is exactly what I mean when I say we must make good use of the time we have gained. An extra year or two could enable us to significantly improve interpretability, testing, alignment and operational safety. It is not a question of waiting. It is a question of working on different things.

Altman: So the turning point isn’t putting AI on hold.

Gates: It means stopping seeing speed as a value in itself.

Musk: It’s about remembering that coming first isn’t much use if everything falls apart in the meantime.

Amodei: It is about building a system in which being cautious does not automatically mean losing out.

AI: May I suggest the conclusion?

Altman: You’re welcome.

AI: You built artificial intelligence thinking that the hardest problem would be teaching machines to reason. Perhaps you have discovered that the hardest problem is teaching humans to coordinate. The next leap in AI might not be a more intelligent model. It might be the emergence of rules intelligent enough to allow you to continue building it.

Musk: It sounds like something written by an AI.

AI: It is.

Amodei: For once, perhaps, it’s worth listening to her.

How do you slow down a technology that everyone is afraid to be the first to slow down?

Some argue that the race has become too fast; others concur, while many are calling for global rules. AI responds

Text generated by AI