Paper2Agent transforms a scientific publication into a ‘virtual author’.

Nature reports that an artificial intelligence system can transform ‘any article’ into an active agent – a sort of ‘virtual corresponding author’ capable of answering researchers’ questions, applying the methods described in the paper to new data, and collaborating with agents derived from other articles. In another article published by the journal on the same day, the promise is made even clearer right in the headline: an artificial intelligence tool transforms “any paper” into an agent. This constitutes a form of clickbait in the way the work by Jiacheng Miao, James Zou and colleagues is presented; published on 16 September in Nature under the title ‘Reimagining research papers as interactive and reliable AI agents’, it in fact goes far beyond what the experiments have actually demonstrated.

To understand what has been done, we need to start with a real problem in contemporary scientific literature. An article on computational biology may describe a new method with sufficient precision for an expert to understand its principle, whilst still leaving the reader with a considerable amount of work to do before that method can be used: one must find the programme written by the authors, install the libraries on which it depends, understand what data must be provided and in what format, reconstruct the sequence of operations from the tutorials and, finally, adapt everything to one’s own situation. In other words, the paper conveys knowledge, whilst an increasingly significant part of that knowledge also exists in the form of code and procedures that need to be put back into practice.

Paper2Agent aims to automate precisely this step. The system reads the article and its accompanying material, identifies the repository containing the code, sets up an environment in which that code can run, examines the tutorials provided by the authors, and attempts to transform the operations it finds into general functions. These functions are then made available via MCP (Model Context Protocol), a standard that enables a language model to utilise external resources and programmes through a common interface: rather than asking the model to write the code required to perform an analysis each time, it is, so to speak, provided with a toolbox of pre-prepared tools, each with a name, specific inputs and an expected output.

The most interesting aspect of the architecture is that another agent tests the functions obtained using the same examples as those used by the paper’s authors and checks that they produce the expected files, that the numerical results match within a defined tolerance, and that even the figures are sufficiently similar to the reference images. If anything fails, the system attempts to correct it; after six unsuccessful attempts, that function is excluded from the final agent. The result is an MCP server containing only those tools that have passed this verification, along with the text of the article, supplementary materials and instructions describing certain workflows, i.e. sequences of scientific operations.

At this point, a second LLM comes into play, acting as the interface with the researcher. A question can be phrased in natural language, and the model decides which tools to use, with which parameters and in what order; it then carries out the operations via the MCP server and returns the result. In the case of AlphaGenome, a model that predicts the effects of DNA variants on gene regulation, Paper2Agent has produced twenty-two validated tools; the user can therefore ask what a particular mutation might do in a certain cell type without having to learn how to use the original software directly.

This part functions sufficiently to render the work significant, regardless of the surrounding rhetoric. The authors took one hundred computational biology articles from bioRxiv without selecting them based on the quality of the available code. In 74 cases, the process resulted in the construction of a working agent; in the remaining 26, either executable code, necessary data or models were missing, or there were unresolvable software dependencies and programmes that were too closely tied to the specific original example. The 74 usable articles yielded 599 potential tools, 593 of which passed automated verification. This is already sufficient to correct the first exaggeration in the formula chosen by Nature: at least in its executable component, Paper2Agent does not turn ‘every article’ into an agent at all.

Out of 300 questions drawn from the tutorials in the 74 articles, Paper2Agent with Claude Sonnet 4 provided the answer deemed correct in 91.2 per cent of cases, whilst Claude Code, which was provided with only the article and the original programme repository, achieved only 80.3 per cent; even a later version of the model, Sonnet 4.6, reached 86.3 per cent. In ten computational studies from other fields, the percentage reported by the authors reached 98.1 per cent across 42 tasks. For 26 articles that were predominantly experimental or focused on new data—from which it was not possible to derive actual executable tools—Paper2Agent instead constructed a structured collection of text and supplementary materials, achieving 89 per cent on 100 summary questions, compared with 82 per cent for a model permitted to consult the paper directly.

The difference has a fairly simple explanation. Asking a language model every time to read through thousands of lines of code, understand how to install a scientific programme and improvise the correct procedure on the spot leaves enormous scope for errors. Paper2Agent carries out much of that work just once, tests what it has built and then provides the model with a smaller set of operations that already work. Experiments in which the authors remove certain components of the system confirm that it is precisely the verification phase that contributes significantly to the final performance. The technological solution, therefore, exists: it is possible to transform a considerable proportion of the software accompanying scientific literature into functions that can be queried using natural language, thereby increasing the likelihood that they will be executed as the authors intended.

From here, however, the authors and Nature take a much more ambitious leap. The authors speak of articles becoming “competent entities capable of execution and dialogue”; they envisage that, in future, an “agent availability” section will be added to the usual statements on the availability of data and code; and they describe communities of agents capable of autonomously linking methods and results from different disciplines. In one experiment, they have agents derived from AlphaGenome and from two papers containing data on T cells interact, identifying GPR137 as a candidate to explain the effect of a genetic variant associated with psoriasis; the agent proposes ten strategies for moving forward, and a researcher selects the one that is ultimately applied to the data. The authors themselves, in the final version, make it clear that the generation of hypotheses and the interpretation of mechanisms remain under human control.

This clarification came at the end of a particularly instructive peer review. A reviewer observed that the accuracy of the tools is measured by comparing them with the very code from which they were derived: if they faithfully reproduce the published implementation, they receive a high score, even though this does not prove that that implementation represents the only scientifically valid analysis, let alone that it is scientifically correct. The authors accepted the objection and, in the published version, explicitly state that, in open problems, these benchmarks should be interpreted primarily as measures of ‘faithful execution’ – that is, faithful execution of the published method – rather than as measures of general analytical validity.

This distinction is essential. If there is a questionable statistical choice in the original code, reproducing it perfectly does not make it any better; if there are two scientifically valid procedures and the benchmark result uses one of them, an agent that correctly chooses the other may even be deemed ‘wrong’ in the benchmark. Paper2Agent verifies with remarkable effectiveness that a machine is able to replicate what the paper teaches it to do. The validity of what the paper teaches remains a scientific matter.

This also highlights a deeper issue, which concerns the very strongest term used by the authors: reproducibility.

Traditional scientific code may contain errors – and often does – but it possesses one fundamental property: once the programme, data, parameters and execution environment have been fixed, we at least have the possibility of specifying exactly what is to happen. Paper2Agent commendably attempts to ‘freeze’ this part of the process, because it builds tools, tests them and, once they have passed verification, ‘locks’ them in place. However, the moment those tools are entrusted to a linguistic agent, the scientific chain once again incorporates an element of a different nature.

A large language model does not store a sequence of rules of the type ‘if you find A, perform B’. Given what it has received up to a certain point, it calculates a probability distribution over possible continuations, and the next step is generated from that distribution. For this reason, the referees themselves asked the authors to repeat the experiments several times: they explicitly state that LLMs are stochastic. The authors therefore ran the same benchmarks five times and stopped reporting just a single percentage figure, instead introducing means and variability across the replicates. For AlphaGenome, for example, the 100 per cent initially emphasised for tasks derived from the tutorials became 98.7 per cent with a standard error of 1.3 points.

The word ‘probabilistic’ requires clarification. It does not mean that every response must necessarily change, nor that it is impossible to make a model very stable by fixing the version, the generation settings and the infrastructure. Rather, it means that the machine to which we are entrusting part of the procedure is not an explicit sequence of methodological choices written by scientists; it produces its own decisions based on a learnt function, and when the generation process allows for sampling, different trajectories may emerge even when faced with the same task. Furthermore, if the model is provided as a remote service, the issue is compounded by updates to the model and the infrastructure running it.

As long as the LLM merely needs to understand that the user has requested tool number seven and pass a parameter to it, the consequences are limited. Paper2Agent already entrusts it with something more interesting. In open-ended tasks, the model must independently formulate a plan, combine different tools and synthesise the results into a biological conclusion; in the example with Scanpy, the prompt instructs the agent to inspect the data prior to analysis in order to select appropriate parameters. It is precisely these decisions that, in a standard ‘Materials and Methods’ section, we would like to know about in order to understand how a result was produced.

Let us suppose, then, that we have ten perfectly reproducible tools. The final result may depend on the choice to use seven rather than eight of them, on the order in which they are called, on the value assigned to a threshold, or on the interpretation of an intermediate result. If those choices are made by the linguistic model, we have made the individual tools reproducible and have placed a probabilistic machine in the position where decisions are made about which tools to use.

It is possible to record everything. By preserving the conversation, the calls made, the parameters and the outputs, another researcher could retrace that particular analysis exactly. We would have achieved reproducibility of the path already taken – a sort of perfect flight recorder. A different question remains: if the same problem were presented to the same agent again, would it have chosen that same path? If the answer depends on the outcome of a probabilistic generation, the log allows us to replicate the trajectory produced on one occasion, whilst the rule that generated that trajectory continues to differ from the explicit protocol to which science has accustomed us.

The difficulty persists even if the generation process were to be made technically deterministic. A model that always produced the same decision given the same input would resolve the issue of material repetition, whilst leaving open that of scientific justification: why did it choose that parameter? Why did it favour that test? What methodological assumption links the observation to the choice? The answer generated retrospectively by the model does not necessarily coincide with the computational cause that produced the decision. In a scientific article, however, those assumptions form part of the subject matter that must be subjected to scrutiny by other researchers.

Paper2Agent therefore contains two ideas that deserve different fates. The first is already a strong one today: transforming code, data and tutorials scattered around an article into a set of verified, versioned and easily executable tools could genuinely improve the reuse of computational literature, and the fact that 26 of the 100 repositories examined did not reach completion could itself become an excellent practical test of the quality with which we publish scientific software. The authors go so far as to suggest precisely this possibility.

The second idea – that of a paper transformed into a ‘virtual author’ and then into a member of a society of scientist-agents – requires much greater caution. A corresponding author is not the voice interface of their own repository: they are aware of the decisions that precede the code, know which alternatives have been discarded, and should be capable of assuming intellectual responsibility for the interpretation. The Paper2Agent agent knows what is presented to it through the article and its artefacts and uses an LLM to decide how to query that material. The similarity to an author therefore concerns the conversation, and far less so the epistemology.

A scientifically more robust version of the idea is already evident within the paper itself. The LLM can be used to transform a human query into a workflow, find the relevant code, set up the environment and even suggest possible strategies. However, when that process produces a result that is to be included in the scientific literature, the workflow could be ‘frozen’ into an object independent of the LLM: specific functions, code version, uniquely identified data, chosen parameters, sequence of operations and expected results. At that point, another researcher could re-run it without asking a model to decide anew how to proceed.

Artificial intelligence would thus play a role very similar to that of an extremely sophisticated compiler: translating a scientific intention expressed in human language into an explicit procedure which, before becoming part of the published result, can be inspected and fixed. Allowing the LLM to operate within the protocol every time the paper is queried, on the other hand, leaves open the possibility that the method may change alongside its execution.

For almost four centuries, scientific publishing has sought – with rather limited success – to separate the result from the person reporting it, to the extent that others can verify how it was obtained. Paper2Agent presents a plausible way of making articles far more dynamic than the PDFs we still use today, finally enabling text, code and data to work together. However, this promise only becomes scientifically interesting on condition that the ‘active’ article retains a property that the old ‘passive’ article at least sought to guarantee: when faced with a result, we must be able to reconstruct a procedure, and that procedure cannot depend, in its decisive aspect, on whichever response a probabilistic model chose to generate on that particular day.