Study finds 44% of cited LLM research claims have expired

Claims about what models cannot do expire fastest. Five of nine reasoning-related findings are judged expired, with none surviving or strengthening. The list covers familiar failings such as the "strawberry" letter-counting problem, sums in unfamiliar number systems, dropped details from the middle of long contexts, and models abandoning right answers under user pressure. The researchers' hunch is that these mistakes have a "correct" outcome, which hands developers a clear training target.

Claims about preferences and tendencies prove more durable. Of the 11 findings that have survived or strengthened, several involve models acting as evaluators, sycophancy toward users, a left-leaning political bias, and near-identical answers to open-ended questions. As a case study, the pair revisited a 2023 test of fabricated scientific citations: GPT-3.5 made up 55 percent of its references, while GPT-6.1 Sol fabricated none and Claude Sonnet 5.5 just 1 percent. Yet Qwen3.7-Flash, also a 2026 model, still fabricated 69 percent in the same test.

The study warns against reading the results as an all-clear. Claude Sonnet 5.5 still fails about 10 percent of two-hop questions, where a model must combine two known facts, down from 43 percent for ChatGPT in the original paper. Results also vary considerably across models even within the same release window: among models released between July and September 2026, expired findings range from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol, and neither release date, provider nor model size reliably predicts the pattern.

Queen and Zou conclude that empirical findings about LLMs should be framed as snapshots of particular models at a particular moment rather than universal rules. They recommend that papers record which model and version was used and when, and release prompts and analysis code so future models can be tested against the same findings.