To rid ourselves of corrupt science, we need even more science

Milena Gabanelli’s Dataroom programme has just reported on how artificial intelligence is being used to cheat in clinical trials. This issue has been circulating for years in laboratories and the editorial offices of scientific journals, and for years it has been brought to the attention of the wider public, both at the meetings organised by the CNR on research integrity and in television and radio programmes where the topic now features with some regularity. I have personally spoken out about these risks at each of these forums, with a persistence that may have seemed excessive to some, and for this very reason I would like to pick up the thread where the news report leaves off – which is, in fact, the point at which the matter becomes interesting.

Fictitious experimental data have always accompanied research, and until recently were recognised as such because the procedure used to fabricate them left identifiable traces. The most widely studied type concerns noise, that is to say, the unintentional variability inherent in every real measurement, arising from the instrument, the sample and the hand of the person working at the bench. Anyone who sits down at a desk to fill in a table of measurements that were never actually taken generates noise that is far too ‘polite’, because our intuitive idea of variability coincides with that of an orderly dispersion around the expected value: values that repeat at short intervals are avoided, as they seem suspicious; the final decimal places are spread out evenly; and the results are kept reasonably close to the mean. Authentic noise is far less disciplined in nature and accumulates improbable coincidences, values that are extremely far removed from the others, and clusters of almost identical results that no forger would dare to transcribe.

Noise is merely one of the traces that the method of production leaves on the final result. Those who duplicate a set of measurements and resize it to suit a second experiment leave sequences that reappear several lines later, whilst those who rely on a spreadsheet to generate data produce distributions that clearly reveal the type of generator used. The reported means may prove arithmetically incompatible with the number of subjects and the scale used to measure them, and the agreement between the experimental data and the theoretical model may appear so perfect as to become implausible, as is the case with the initial characteristics of the arms in a randomised trial when they resemble one another more closely than a random draw would suggest. In short, every shortcut taken during the fabrication process leaves a statistical artefact somewhere within the figures, and it is upon this phenomenology that the edifice of post-publication scrutiny has been built over the last thirty years – a process that hunts down the fingerprint left by the forger’s methods far more than the content of the forgery itself.

A generative model does not take any of those shortcuts, because it samples directly from the pattern that real data has shown it millions of times during training, and produces tables featuring credible noise, consistent internal correlations, and missing values distributed as they would be in the real-life setting of a hospital ward, alongside sequencing reads, mass spectra, electronic health records populated with patients who were never born. The forgery emerges devoid of the tell-tale signs we were looking for and equipped with just the right imperfections, and the very foundations upon which our verification tools rested crumble just as the cost of production plummets to a few dozen euros per item. The falsification of scientific images using generative AI—which I documented last year with Angelo Parini in the American Journal of Hematology—offers only the most striking example of a trend affecting the entire chain of experimental evidence, from the raw data output by the instrument to the summary table that reaches the reviewer’s eyes.

The race between those who fabricate data and those who expose it is, moreover, a foregone conclusion, for the structural reason that every whistleblower becomes training material for the next generation of fraudsters. Asking the scientific community to chase after fraudulent data means condemning it to lose ground with every model update, with the depressing consequence that the only guaranteed result will be a constant stream of scandals to report.

The approach now being adopted in the clinical setting turns the issue on its head and certifies the original data at the exact moment it is produced. In regulated research, this practice is already standard: the system records data via an unalterable audit trail; the raw data is generated already signed and time-stamped; the inspector traces it back to the source and verifies the chain of custody leading from the patient’s record to the published table. Bringing the same architecture into the preclinical laboratory is a matter of straightforward engineering, given that the equipment used there can sign the file upon acquisition just as naturally as it currently records the date and parameters on it, and a journal can require, alongside the manuscript, the cryptographic fingerprint of the raw data from which each table is derived. At that point, fabrication would require the falsification of the chain of custody, which is an act of a completely different nature, one that is traceable and punishable under criminal law.

All this, however, remains a band-aid applied to an organism that continues to produce the disease, because the root of the problem lies elsewhere and we have known about it for decades. The demand for fake articles is generated by an evaluation system that measures people by counting their output, and which has turned the number of publications and the journal hosting them into the currency with which competitions, promotions and funding are bought. The San Francisco Declaration on Research Evaluation has been denouncing this drift since 2012; universities sign it enthusiastically yet continue to compile their internal rankings in exactly the same way, whilst entire national systems have, by law, tied academic careers to numerical thresholds that no committee can ignore. As long as researchers face the choice between publishing ten mediocre papers and remaining stagnant with two serious ones, the ‘paper mill’ industry will have a stable and growing clientele, and any technical barrier will merely serve to drive up the price of the forgery.

These are two measures that only make sense when taken together, because data certification makes fraud difficult, whilst the reform of the evaluation system renders it pointless. Neither is particularly suited to a front-page headline, and this is perhaps why the public debate touches on them only in passing, before returning to the story of the individual case – the professor who has been exposed, the journal forced to retract the article.

Investigative journalism serves a useful purpose when it brings the issue back to those who have the means to tackle it. In this area, the news has been circulating for some time and, with every new report, produces the same wave of outrage, which invariably recedes, leaving the shore exactly as it was; and the risk, when we stop at mere indignation, is that we hand those who are already sceptical of medicine yet another argument, whilst leaving the machine that churns out false data intact, along with the incentives that fuel it.

Changing those incentives, and making those who cheat pay a price proportionate to the damage they have caused, are tasks that fall to society as a whole, because no laboratory can rewrite the rules of competitions and no journal has the power to impose a penalty. Each of these decisions, however, rests on technical evidence capable of standing up to an appeal or a court of law, and such evidence can only be established by genuine science, through forensic analysis of the figures and the chain of custody linking the raw data to the published table. The conclusion may appear paradoxical only to those observing the matter from afar: to rid ourselves of corrupt science, we need more science – in far greater quantity and of far higher quality than we currently devote to the problem.