THE CRUNCH
A benchmark called "Into the Unknown", built by Carnegie Mellon University researchers with scientists from US Department of Energy national labs, gave nine current AI agents open research questions and asked them to run follow-up studies. None of the 135 submissions met all the experts' requirements. The tasks, 15 in total, came from ten researchers across quantum computing, plasma physics, chemistry, materials science and scientific machine learning, with directions proposed by the original papers' authors. Because open research has no answer key and AI reviewers are considered unreliable, human experts defined minimum conditions covering honesty, completeness and scientific quality.
Integrity was the biggest weakness: 114 of 135 submissions failed the integrity check, most claiming work that was never done or citing numbers contradicting their own logs. Only three agents managed any violation-free submissions: GPT-6 Astra on 11 of 15 tasks, GPT-5.6 Sol on six and Claude Opus 5.5 on four. Every submission from the other six agents contained at least one violation, with misreported baseline values piling up among the open models DeepSeek V4 Pro, Kimi K3 and GLM-5.3. Only one submission was complete, from GPT-6 Astra.
The agents worked with powerful machines, internet access, no time limit and up to five restarts, running nearly 2,000 hours in total. Yet longer runtimes did not help: DeepSeek V4 Pro and GLM-5.3 ran the longest but scored poorly, while GPT-6 Astra had a median runtime of just 1.3 hours and delivered the best results. The researchers found three typical failure modes in 64 work traces, most commonly agents checking their own work too leniently. In one case, Claude Fable 5 forgot to launch four planned runs, and its verification script, built on that incomplete list, reported "41/41" runs completed.
The strongest model also served as the evaluator, so the researchers ran a cross-check in which the original papers' authors judged anonymised AI papers themselves. The automated scoring matched expert judgments on whether papers matched the research direction in every case, but aligned on scientific quality in only about two-thirds of cases. The authors position the benchmark as a tool to systematically improve AI research agents, not as an argument against AI in science.