From AutoResearch to Proof-of-Improvement: Decentralising Optimisation and Discovery

Autonomous research systems, such as Andrej Karpathy’s autoresearch, demonstrate a simple yet powerful paradigm for iterative model improvement. In this framework, an agent modifies a training script, executes short experiments, evaluates performance against a defined metric, and retains only those changes that yield measurable improvements. While minimal in implementation, this structure captures an important computational asymmetry: discovering improvements is often resource-intensive, whereas verifying the quality of a given improvement is comparatively inexpensive.

This asymmetry is not unique to machine learning. It closely resembles classes of problems I studied in operations research, where identifying optimal or near-optimal solutions is computationally difficult, but validating candidate solutions is straightforward. For example, in supply chain optimisation, determining optimal routing, inventory allocation, and replenishment strategies across a network is complex; however, evaluating a proposed solution in terms of cost, service levels, and constraint satisfaction is deterministic. Similarly, in scheduling problems, constructing feasible schedules across multiple constraints is challenging, whereas verifying feasibility and computing objective values is relatively trivial. In financial optimisation, exploring portfolio allocations under constraints is computationally intensive, but verifying risk, return, and compliance with constraints is direct.

The autoresearch framework effectively operationalises this asymmetry in the context of machine learning by structuring the problem as a search over program space – encompassing code, hyperparameters, and architectures – paired with deterministic evaluation. Only those candidate modifications that improve the objective function are retained, creating a localised process of iterative optimisation.

This structure suggests a natural extension beyond a single execution environment. Rather than a single agent performing sequential experimentation, one may consider a distributed setting in which multiple independent agents propose candidate modifications, a shared evaluation function is applied, and results are verified by independent parties. In such a system, accepted improvements can be recorded in a shared state, transforming the process from a local loop into a coordinated protocol.

Under this formulation, the system begins to resemble a distributed ledger, albeit with a distinct objective. Instead of ordering financial transactions, the system orders improvements. Each accepted result constitutes a commit that references a prior state, incorporates a defined modification (e.g. a code change and resulting model checkpoint), and is associated with a measurable change in performance. Unlike linear blockchains, however, the structure is more naturally represented as a directed acyclic graph (DAG), where multiple branches of experimentation coexist, and superior results propagate through selection mechanisms.

The central primitive in such a system may be described not as Proof of Work or Proof of Stake, but as Proof-of-Improvement. A submission is considered valid if it demonstrably improves a predefined metric, can be reproduced under specified conditions, and is evaluated deterministically within acceptable tolerances. This replaces traditional hash-based proofs with proofs grounded in useful computation.

This reframes discovery as a consensus problem: not agreeing on state, but agreeing on progress.

To operate in an environment of untrusted participants, the system can be decomposed into three functional layers:

• First, an execution layer, in which agents perform experiments and generate candidate improvements off-chain;
• Second, a verification layer, where submitted results are independently re-evaluated to confirm claimed performance;
• Third, a coordination layer, in which commitments, such as hashes of code, datasets, and model artefacts, are recorded, experiment lineage is tracked, and rewards or attribution are assigned. The use of content-addressed artefacts ensures integrity and traceability across all components;

A critical aspect of this design is the incentivisation of the verification layer. While verification is computationally cheaper than discovery, it is not costless, and therefore requires explicit economic alignment. Validators may be required to stake resources when attesting to the correctness of a submission, receiving rewards for accurate validation and incurring penalties in cases of incorrect or fraudulent attestations. Similarly, mechanisms for third-party challenge can be introduced, whereby participants are rewarded for identifying invalid results. In this way, verification is transformed into an economically secured process, ensuring that truthful validation is the dominant strategy even in the presence of untrusted participants.

The principal challenges in such a system are not computational but structural. These include preventing metric manipulation and overfitting, maintaining the integrity of evaluation datasets (including the use of hidden or rotating test sets), managing non-determinism in execution environments, coordinating validation without centralised trust, and controlling the expansion and convergence of the experimental search graph. These challenges are analogous to those encountered in large-scale optimisation and distributed systems, where incentives, constraints, and feedback loops must be carefully designed to ensure stability and efficiency.

An additional extension of this framework involves the application of zero-knowledge techniques. In this context, participants could, in principle, provide cryptographic proofs that a computation was executed correctly without revealing the full details of the computation or underlying data. For example, it may be possible to prove that a model evaluation was conducted on a committed dataset or that a reported metric corresponds to a specific code and input configuration. While such techniques remain computationally intensive for full training processes, they are increasingly applicable to evaluation steps and constrained verification tasks, thereby reducing trust assumptions in specific components of the system.

Overall, this approach reframes model development as a distributed search process over program space, coordinated through reproducibility, cryptographic commitments, and incentive mechanisms. Experiments function as transactions, improvements as consensus, and computational effort as a form of stake. Systems such as autoresearch provide a minimal illustration of this paradigm; however, the underlying concept is more general. When extended into a distributed setting, it yields a protocol for coordinating discovery itself, aligning closely with established principles in operations research and large-scale optimisation.

The implication is that intelligence is no longer solely engineered within institutions, but coordinated across a distributed system – where progress itself becomes a matter of consensus.