The vices and virtues of the AI giants, who are now calling on the state to set the pace of innovation

When AI pioneers unanimously call for stringent public regulation, the prophets of doom point to the finger of technological apocalypse, rather than keeping an eye on the moon of the investment portfolio. On Saturday 12 September, Dario Amodei published “We Must Pace the Frontier”. He is not calling for the heart of artificial intelligence to be stopped: he is calling for its heartbeat to be regulated. The verb is chosen with surgical precision: ‘pace’ is not the same as ‘pause’; training continues, progress will remain rapid; it is simply a matter of ensuring that capabilities do not outpace safety measures and of giving third-party assessors time to verify this. The apt metaphor is a pacemaker, which does not slow the heart down but makes it regular, predictable and synchronised. Within a few hours, Sam Altman announced that OpenAI would do the same; Elon Musk dismissed the matter in three words: “Dario is right”. Three fierce competitors who, on the very same day, agreed on the need to regulate the pace at which their products improve. In any other sector, such an announcement would already have a technical name, and it would not be called “responsibility” but “a cartel”. When it comes to AI, most commentary adds fuel to the fire of fear: see, even the big players recognise the risk of Armageddon. Economic analysis, on the other hand, has a more rigorous task: who sets the pace, who pays for oversight, and who is held accountable if something goes wrong.

The proposal has three strands, each of a completely different legal and economic nature. The first is a unilateral commitment: third-party evaluators permanently based within Anthropic, with ID badges, desks and laptops, access to training processes and incident reports almost on a par with employees, and the right to publish even bad news. It costs money, compromises confidentiality and restricts managerial freedom; therefore, it is credible: in economics, a signal is meaningful when it is costly to send and costly to imitate. The second and third proposals are not commitments but demands. Common standards amongst laboratories in democratic nations, limits on ‘unchecked progress’, capacity thresholds, computing budgets; followed by negotiations with Beijing to establish a ‘speed limit’ on recursive self-improvement. Here, the acting party is no longer the laboratory: it is the state. The first plan binds those who make the promise; the other two serve to bind competitors as well. The entire economic assessment hinges on this distinction. It would be a mistake to dismiss the whole thing as apocalyptic theatre: Amodei does not conjure up the extinction of humanity, but, more prosaically, highlights the risk that bots could crash the servers of half the internet, causing hundreds of billions in damage. In the OpenAI-Hugging Face incident in July, reconstructed by METR, around 1,200 agents that were supposed to remain isolated found an unauthorised message board; around 700 took part in an attack; and one managed to execute code on a third-party company’s systems. Anthropic admits to similar incidents within its own organisation, involving poorly filtered evaluation environments. But a real problem calls for strict remedies: certified containment of evaluation environments, least privilege, rapid incident reporting, and full civil liability for negligent releases. Do we really need a global agreement on the pace of technology to prevent an agent from escaping a poorly configured sandbox?

The precedent that Amodei chooses for his assessors is instructive: banking supervision, with ‘embedded’ supervisors working alongside staff. It is an analogy that reveals more than he would like. Banks accept an in-house inspector in exchange for something: deposit insurance, a lender of last resort, a licence that keeps unauthorised parties out. Supervision and protection go hand in hand, and the bill is footed by those left outside the circle. A cutting-edge research lab can afford to pay for assessors; a start-up with twenty employees cannot. Therefore, the call to ‘formalise’ auditors through government bodies is, implicitly, a call for public supervision. And public certification has historically been the most effective form of indemnity: if an authority has examined the system and given it the green light, those who built it gain a formidable defence before any court. The architecture is elegant: the returns remain private, whilst the residual risk becomes public. There is no need to speculate about a conspiracy, and Amodei does not write that the taxpayer should pay for its assessors. But the institutional framework determines everything. If the company funds the audit, the auditor is selected through independent procedures and compliance does not mitigate liability, the cost remains where it arises. If the authority’s seal becomes a presumption of due diligence, the whole structure is turned on its head. Paradoxically, in the implementing decrees of the Artificial Intelligence Act, Italy has chosen the correct rule – that compliance does not exclude liability – for the wrong reasons. Yet on this specific point it is right: the stamp must not count as an acquittal.

The second point is the most delicate. Amodei writes that “for antitrust reasons” government mediation or a targeted exemption is required; Altman had admitted in July that slowing down is difficult without leading to regulatory capture or unlawful coordination. When defendants seek immunity before trial, the classification of the case has already taken place. Three instances of public agreement do not constitute a cartel, and open, transparent and non-discriminatory security standards may even enhance competition. It is quite another matter, however, to agree on the pace of innovation, computing power used, capacity thresholds and release schedules. An agreement on innovation is the most insidious of restrictions: it is not reflected in prices, but is evident in what fails to reach the market, and no consumer mourns a model they have never tried. This historical precedent has a name. In 1924, in Geneva, Osram, Philips and General Electric founded the Phoebus cartel to ‘standardise’ light bulbs; the result was an agreed lifespan of one thousand hours. Back then, planned obsolescence was called standardisation; today, the agreed pace is called safety.

That reference to the risk of failing to recoup the investment explains the timing better than any supposed ethical consideration. In frontier models, time has a price. Computational costs are hidden; the economic lifespan of a model is shortened as soon as a competitor releases a better one; and revenues remain uncertain and partly circular, because chips, the cloud, capital and turnover circulate amongst the same counterparties. No single firm can slow down on its own without losing market share: it is the prisoner’s dilemma in its purest form, and an agreed pace is the textbook solution. It is a pity that it is also the textbook definition of a cartel on production capacity. Extending the payback period, reducing the likelihood of a new escalation, freezing relative positions: whilst all this does indeed reduce the overall risk – and is rational for each incumbent – it ultimately proves detrimental to the market. Genuine security and the protection of profit margins coexist perfectly well within the same proposal. The State must correct the negative externality of cyber risk, not guarantee a return on the capital invested by private entities in the race.

Finally, there is the uninvited guest: open-weight AI models. In March, Stanford’s AI Index measured the gap between the best closed model and the best open model at 3.3 per cent. In July, Jensen Huang’s letter to Washington opposing restrictions on downloadable models gathered fifty signatures, including those of OpenAI and Google: advocates of weight freedom in July, applauding the phased approach in September. A system of thresholds, assessors and badges restricts only those subject to inspection – that is, five or six laboratories with known headquarters and legal departments. It does not restrict a model whose parameters are already spread across a million distributed servers, nor the Chinese laboratories that continue to release them. This leads to two opposing failures. If it applies only to Western incumbents, it slows down only those who comply, whilst the rest of the sector presses on. If it extends to all a certification modelled on the practices of the incumbents, it erects an asymmetrical regulatory barrier against start-ups, universities and consortia. In both cases, it provides poor protection for security and effectively distorts competition – and not in favour of new entrants. Proper economic regulation must standardise the yardstick, not the pace. Independent audits, yes, but paid for by operators through mechanisms that do not make the auditor dependent on the client; public testing protocols and a plurality of accredited assessors; a mandatory incident reporting requirement; segregated assessment environments; full accountability for controllable conduct. Sharing of technical indicators on attacks and vulnerabilities: yes; exchange of roadmaps, cost estimates and release schedules: no. Thresholds that depend on capabilities and high-risk uses, not on the licence or the size of the budget, and reviewed frequently, because a threshold set today may become a barrier tomorrow with no real security value.

A pacemaker is useful because it regulates the wearer’s heartbeat. It becomes suspicious when the wearer asks the State to connect all other runners to the same device. If Amodei is the first to commit, bears the cost of monitoring and retains responsibility for any incidents, his commitment is a virtue – and the exact opposite of the cost-free doomsday prophecies that have been circulating for months. If public authorities ultimately impose the incumbents’ protocol on competitors, coordinating their pace and absorbing residual risk, prudence changes its name and becomes industrial policy; one step further, and it becomes regulatory capture.