TasteVal Brings Research Judgment into the RSI Debate

Executive Summary

Choosing better experiments could make AI-assisted research faster even without adding GPUs. TasteVal puts a number on that possibility: its authors report that their best model reaches expert-level results with a 2.30× experimental-compute advantage. The consequential discourse shift is Jack Clark’s response: in today’s Import AI, he moves his expectation for when full recursive self-improvement might become possible from 2028 to 2027.

The evidence supports a narrower conclusion than that forecast. TasteVal measures experimental judgment on assigned research problems, not the ability to choose a research agenda or autonomously improve a frontier system. Its fastest reported progress also depends on which metric you use.

What TasteVal Actually Tests

Delayed discovery: the paper was submitted October 5 and revised October 6. Today’s development is Clark’s interpretation, not a new benchmark release.

P-Zero Research’s study overview describes eight tasks spanning training-data curation, pretraining and fine-tuning, preference modeling, and adversarial robustness. The evaluated model designs experiments and interprets results; a fixed coding agent implements and runs them. Each run stops after 40 H100 GPU-hours or 120 wall-clock hours. That separation is important: the benchmark tries to distinguish deciding what to test from being good at writing the implementation.

The human baseline takes the best expert attempt on each task from a pool of 24 experts, with at least two per task. The study evaluates 20 models released between 2023 and 2026.

“Research taste” is operationalized as experimental-compute efficiency. If a model reaches the expert score using half the experimental compute, it receives a 2× multiplier. This is not a claim that it is twice as creative, twice as accurate, or twice as capable across science.

The Acceleration Is Metric-Dependent

The authors report a 2.30× multiplier for Opus 5.5, with a 95% confidence interval of 1.15–4.37×, at roughly one-thirtieth of the human baseliners’ average cost per attempt. That is a substantial reported advantage, but the experimental-compute multiplier and monetary-cost comparison are different quantities.

Their headline trend is striking: frontier models’ multiplier doubled approximately every three months after December 2025, compared with every 14 months beforehand. But the paper’s abstract supplies an essential qualification: final normalized performance shows no trend break, doubling every 14.6 months.

Both observations can coexist. Reaching a target score with fewer experiments is different from achieving a higher score by the end of a fixed budget. For research automation, efficiency is consequential because experiments consume scarce compute. For forecasting, however, “research judgment is accelerating” needs that metric attached; the result does not establish an equally sharp acceleration in every measure of research quality.

From Useful Researcher to Self-Improvement Loop

Clark treats creativity as a potential bottleneck to recursive self-improvement: systems that can implement experiments still need productive ideas about which experiments to run. TasteVal gives that argument more concrete evidence than another coding demonstration. His earlier timeline remains an interpretation of the evidence, not a measured outcome.

The authors themselves describe their timeline exercise as a naive extrapolation rather than a forecast. Holding the measured improvement relationship constant moves the AI Futures Model’s median superintelligence date from mid-2030 to late 2028. That conditional calculation should not be confused with Clark’s separate judgment about when self-improvement becomes possible.

The remaining limits are material. Tasks are cheap and fast to verify, potentially favoring capabilities that labs can improve most readily. The human baseline does not include top researchers. Most importantly, TasteVal does not measure choosing the most fruitful problems. The tasks are withheld to prevent contamination, which also limits public inspection and replication.

This sharpens the developing view of agents as combinations of capabilities rather than a single autonomy score. Implementation, experimental selection, interpretation, and agenda-setting deserve separate tests. TasteVal strengthens the case for useful automated experimental judgment; it leaves the leap to a self-sustaining research loop unresolved.

Further Reading

• TasteVal paper: preserve the distinction between compute efficiency and final performance when reading the growth claims.
• P-Zero’s overview: a concise account of the setup, limitations, and explicitly conditional timeline extrapolation.