type
Post
status
Published
date
Nov 15, 2026
slug
summary
tags
category
技术分享
icon
password
The Hardest Benchmark Yet: Why Brute-Force AI Reasoning Fails at Predicting Wet Lab Results
A new evaluation framework reveals that increasing computational power and reasoning intensity is insufficient for accurate wet experiment prediction. Led by UniPat AI and a consortium of top academic and industrial partners, the Science Prediction Universe (SPU) benchmark tests whether artificial intelligence can reliably forecast experimental outcomes before physical testing begins.
The Challenge of Wet Experiment Prediction
In scientific research, predicting the results of "wet" experiments—those involving physical chemical or biological processes—before execution is a critical step. It allows researchers to filter promising directions and save the significant time and financial costs associated with repeated trial-and-error. While AI models are becoming increasingly capable of autonomous research tasks, such as literature review and software operation, their ability to predict physical experimental outcomes has remained difficult to quantify.
Existing benchmarks like BrainBench or ScienceAgentBench focus on data analysis or direct judgment of known results. SPU takes a different approach. It combines autonomous research processes with wet experiment prediction. The model must not only predict outcomes but also demonstrate the scientific tool usage and long-horizon task execution required to arrive at those predictions.
Structure and Rigor of the SPU Benchmark
The SPU benchmark consists of 116 tasks covering 4,207 wet experiments across chemistry, materials science, biology, and medicine. These tasks are derived from 85 recent public papers and four sets of unpublished collaborative experiments. To prevent memorization or contamination, results for these experiments are hidden from the model during the evaluation process.
The evaluation environment is strictly controlled. Models are given 8 hours per run, equipped with 8-core CPUs, 16 GiB of memory, and a single H20 GPU. They are allowed to use internet search and install scientific software. However, the final scoring is conducted in an isolated, offline container to ensure integrity. The system uses the Spearman correlation coefficient to measure how closely the model's predicted ranking of experimental results aligns with the actual measured ranking.
To account for luck, SPU generates 4,000 random rankings for each task to establish a baseline. A model's performance is only considered valid if it significantly exceeds this random threshold, ensuring that the results reflect genuine scientific reasoning rather than chance.
Disappointing Results for Frontier Models
The initial evaluation covered 12 frontier models in 13 different configurations. The results were underwhelming. Even the top-performing model, Claude Opus 5.5, achieved an average score of only 0.486. Across all configurations, only 34% to 52% of tasks exceeded the random baseline. This indicates that current AI systems are still far from providing reliable predictions for complex wet experiments.
Analysis revealed two distinct groups of models: a "leading group" and a "lagging group." While the leading group outperformed the lagging group by approximately 0.15 points on average, both groups struggled significantly. A key insight from the study is that prediction errors often occur at the stage of selecting judgment criteria. For instance, in a polymerization task, one model focused on solvent solubility while another focused on catalyst compatibility. The model that failed to adjust its reasoning based on new evidence produced incorrect rankings, highlighting that simply identifying issues is not enough; the AI must effectively integrate that information to change its prediction logic.
Inefficiency of Increased Compute and Reasoning
A counter-intuitive finding emerged regarding computational resources. Common assumption holds that increasing reasoning intensity or compute time would improve accuracy. However, SPU data showed that for GPT-6 Astra, increasing the reasoning strength to maximum resulted in a median runtime increase of 4.8x and an API cost increase of 5.3x, yet the prediction scores remained almost unchanged.
Reviewing the logs revealed that the model often finalized its answer early in the process and then spent the remaining time on minor adjustments that did not impact the final ranking. Similarly, using more complex tools, such as Monte Carlo sampling, did not necessarily improve results if the tools were not applied to the most critical aspects of the judgment. This suggests that brute-force computation is inefficient for this task; instead, targeted resource allocation based on specific scientific insights is required.
Implications for AI-Driven Science
The SPU benchmark demonstrates that "AI for Science" has reached a plateau where raw capability is no longer the sole limiting factor. The gap between current AI models and reliable wet experiment prediction is significant. Researchers are advised to use such benchmarks to evaluate specific models for specific domains before deploying them in actual research workflows.
The results emphasize that AI scientific reasoning is not just about processing power or general knowledge, but about the nuanced selection of experimental variables and the ability to adapt hypotheses in real-time. Future improvements in AI-driven scientific discovery will likely depend on refining these reasoning pathways rather than simply scaling up hardware.
原文出处:量子位,https://www.qbitai.com/2026/11/510232.html
- 作者:数字Baby
- 链接:myaigc.top/article/419b7c2d-0e98-81e2-bfbe-f6dca938917a
- 声明:本文采用 CC BY-NC-SA 4.0 许可协议,转载请注明出处。




