New Benchmark Reveals AI Struggles to Predict Wet Lab Results Despite High Compute Resources

type

Post

status

Published

date

Nov 14, 2026

slug

summary

New Benchmark Reveals AI Struggles to Predict Wet Lab Results Despite High Compute Resources

A new evaluation framework named Science Prediction Universe (SPU), led by UniPat AI and developed with input from top academic and industrial institutions including Stanford, Harvard, and Cornell, has highlighted a significant gap between current large language model capabilities and reliable wet-lab prediction. The benchmark focuses on a high-value scientific task: predicting the outcomes of physical experiments before they are conducted.

The Case for Wet Lab Prediction

As AI for Science tools become more autonomous, capable of reading literature and running software for extended periods, the focus is shifting to the next stage of research: can AI make accurate scientific judgments? Predicting wet lab results is crucial because physical experiments are expensive and time-consuming. In chemistry, this involves setting up reaction conditions and separating products; in biology, it requires lengthy protein expression processes. A correct prediction allows researchers to filter promising directions, saving resources and accelerating discovery.

SPU differs from existing benchmarks like BrainBench or Terminal-Bench 4.0. While those tests focus on data analysis or task completion, SPU emphasizes the model's ability to autonomously research and predict experimental outcomes. The test includes 116 tasks covering 4,207 wet experiments across chemistry, materials science, biology, and medicine. The task set comprises 100 public tasks derived from recent papers and 16 private tasks from unpublished collaborative experiments, ensuring the models cannot simply memorize the answers.

Methodology and Rigor

The evaluation process is strictly controlled. Each model run is limited to 8 hours with specific hardware constraints: an 8-core CPU, 16 GiB of memory, and one H20 GPU. Models are allowed internet access for literature retrieval and installing scientific software, but the scoring process occurs in isolated, offline containers to prevent data contamination. Predictions are compared against actual results using the Spearman correlation coefficient. To account for luck, SPU generates 4,000 random permutations for each task to establish a baseline; a model must significantly exceed the 95th percentile of this random threshold to be considered a valid prediction.

Model Performance: A Disappointing Reality

The initial results from 12 frontier models and 13 configurations reveal that current AI systems are far from reliable for this task. Even the top performer, Claude Opus 5.5, achieved an average score of only 0.486 across all scored tasks. Most configurations only outperformed the random baseline in 34% to 52% of the tasks. The models were divided into two tiers: a leading group of six and a lagging group of seven, with the top group scoring approximately 0.15 points higher on average.

Analysis of specific tasks, such as a polymerization reaction with 12 different solvents, showed that errors often stem from the models selecting the wrong variables to prioritize. For instance, while some models correctly identified catalyst compatibility, others focused on irrelevant metrics like solvent solubility, leading to incorrect rankings. The ability to adjust reasoning based on new evidence remains a weak point for these systems.

The Diminishing Returns of Increased Compute

A counter-intuitive finding from the SPU results challenges the assumption that more reasoning power always leads to better scientific predictions. When the inference level of GPT-6 Astra was increased to its maximum setting, the median runtime increased by 4.8 times and the API cost by 5.3 times, yet the prediction scores remained nearly unchanged. Model logs indicated that at high inference levels, answers were often finalized at 26% of the total runtime, with subsequent calculations providing only minor tweaks that did not improve accuracy. Similarly, increasing the use of scientific tools, such as Monte Carlo sampling, did not significantly boost performance if the tools were not applied to the core decision-making process.

Implications for Research

These findings suggest that simply throwing more compute or time at a problem is not an effective strategy for improving wet-lab prediction. Instead, the focus should be on optimizing how models select judgment criteria and allocate resources. The results also highlight that model performance varies significantly depending on the specific measurement target and experimental conditions; a model that excels at predicting ion conductivity may fail at predicting electrochemical windows. Researchers are advised to validate models on specific task types using SPU-like benchmarks before deploying them in formal experimental workflows.

Source: 量子位, https://www.qbitai.com/2026/11/510232.html