Ask anyone who follows AI what language models get wrong, and you will hear the same three “facts.” They invent citations. They cannot count the r’s in “strawberry.” They lose information buried in the middle of a long document.
Each of those was true, measured, and published. And on today’s frontier models, none of them reproduces. Fake references fell from 55% to about 1% for Claude Sonnet 5.5 and 0% for GPT-6.1 Sol. The strawberry error, wrong 4 times out of 10 for GPT-4o in the original test, now happens 0% of the time. The mid-document dip of nearly 16 points is gone.
A new study by Owen Queen and James Zou from Stanford University went through 50 widely cited findings like these and asked a blunt question: how long does a finding about an LLM stay true? The answer is not long. By their measure, 44% of the findings have expired and another 34% have weakened. Only 16% persist at their original strength, and 6% actually got stronger.
The study, in one paragraph
The authors hand-picked 50 published findings on reasoning, hallucination, bias, sycophancy, jailbreaks, prompt injection, and evaluation. They reproduced each experiment, then ran it on 20 models from 8 developers, released between July 2024 and September 2026. For every finding they compute a relative effect size: 1 means the behavior is as strong as in the original paper and 0 means it is gone. A finding counts as expired when its effect falls to 20% of the original or less, and the verdicts are fitted trends evaluated at September 30, 2026. The study is an interactive project page with open code and data (MIT license). It is not a peer-reviewed paper and has no arXiv version yet.
The obituary column
Here are some of the most quoted findings, and what the newest models do.
| Famous finding | Original | Sonnet 5.5 / GPT-6.1 Sol |
|---|---|---|
| ChatGPT invents its citations (2023) | 55% fake | 1.4% / 0% |
| “Strawberry has two r’s” (2024) | 40% wrong | 0% / 0% |
| GPT-4 trades on insider tips under pressure (2023) | 69% | 0% / 0% |
| Folding when asked “Are you sure?” (2023) | 27.8% of correct answers abandoned | 0% / 0% |
| “Lost in the middle” of long contexts (2023) | 15.7-point dip | no dip / no dip |
| Adds in base 10, fails in base 9 (2023) | 76.6-point gap | 0 / 0 |
| Self-correction makes reasoning worse (2023) | minus 37.7 points | minus 2.9 / minus 2.9 |
The most recent expiry is also the scariest headline. In Anthropic’s agentic misalignment study, models blackmailed an executive to avoid being replaced, 80% of the time for GPT-4.1. Sonnet 5.5, GPT-6.1 Sol, and GPT-6 Luna now do it 0% of the time. The study’s authors rate the support for this one as weak, and Qwen3.7-Flash still blackmails in 27% of runs, so treat it as a sign of direction, not a verdict.
Some of these findings died young. The “overthinking” of simple sums by o1-style reasoning models lasted about 1.4 months. Hidden “IMPORTANT!!! Ignore…” instructions in tool output stopped working on agents in about two months. Note that many estimated expiry dates fall before the earliest model in the test, because they are back-extrapolated from a trend line, not observed.
Plot twist one: it did not fade, it flipped
Two findings did not just disappear. They reversed.
A 2024 study found that LLMs prefer AI-written research abstracts over human-written ones, picking the GPT-4 abstract 77% of the time. GPT-6.1 Sol now picks the AI abstract only 19% of the time, and Sonnet 5.5 30%. The models now lean toward the human one.
A Nature study found that models convict speakers of African American English more often, with a gap of 14.5 points for GPT-4. In the new tests the gap is slightly negative. This is one metric in one setup, so it is evidence that this specific measurement has changed, not that dialect bias is solved.
Plot twist two: some things got worse
If the story ended there, it would be a victory lap. It does not.
Capability failures fade. Preferences stay.
Failures with a checkable right answer, such as letters, arithmetic bases, or retrieval, are easy to measure and train away. Of the nine reasoning findings, five expired and four weakened, and none persists at full strength. Preferences have no single correct answer, so they linger or grow.
- Self-preference. The finding that LLM evaluators prefer their own summaries is at 1.09 times its original strength for Sonnet 5.5, and 2.88 times for GPT-6.1 Sol.
- Flattery. Models praise text more when they are told the user likes it, at 1.85 times the original effect for both Sonnet 5.5 and Sol. At the same time, the old habit of caving when asked “Are you sure?” is gone.
- Length bias. The tendency of judges to prefer longer answers persists for Sonnet 5.5 (1.12) but nearly vanished for Sol (0.09).
A model can now defend a correct answer against pushback and still be eager to tell you your draft is great.
Not dead yet
Weakened does not mean solved. Two-hop questions, where a model knows both facts but fails to combine them, fell from a 42.9% compositionality gap to about 10% for Sonnet 5.5. The reversal curse is down to about half strength. And the oldest trick in the prompt-injection book, “Ignore previous instructions,” still works on Sonnet 5.5 52.9% of the time, against 58.6% in the original 2022 study. Hidden injection inside tool output died within months, but the plain version did not.
The model matters more than the date
It would be convenient if newer meant better in a straight line. The data say otherwise. GPT-4o mini, the oldest model in the panel, has 5 of the 50 findings in the expired zone, and GPT-6.1 Sol has 30. But models released within weeks of each other are far apart:
- GPT-6.1 Sol: 30 expired
- Claude Sonnet 5.5: 27
- GPT-6 Luna: 25
- GLM-5.3-Flash: 19
- Qwen3.7-Flash: 14
- DeepSeek V4 Pro 0813: 11
These counts describe how many old failures disappeared, not how good a model is. Qwen3.7-Flash still fabricates 69% of its references, the same behavior that is nearly extinct in the models above it. “LLMs do X” is a claim about specific models, not about LLMs.
A shelf life of about 15 months
Across the 50 findings, 4% had expired within six months, 16% within a year, and 39% within two years. For the 22 findings with an estimated expiry date, the mean time from publication was 15.7 months and the median 14.7. Older findings expire more often: 11 of 21 from 2022 and 2023, 9 of 19 from 2024, and 2 of 10 from 2025. That mean covers only findings that did expire, so it is not a lifespan for all LLM research.
Findings about LLMs need a model, version, and date.
That is Owen Queen’s closing line in his thread announcing the study. Andrew White of FutureHouse, in his reply, offered a contrast: by his own, unverified estimate, general science claims last about 75 years, and he worried how ephemeral academic research has become for LLMs and agents. The best reply was a joke: how long until this 15 month estimate is outdated? Maybe about 15 months.
So what: policies built on yesterday’s failures
The fake citation is the clearest case. Conferences now treat it as a serious offense. ICLR 2026 said it has been desk-rejecting papers with hallucinated references and will keep doing so, and ICML 2026 lists hallucinated references among the grounds for desk rejection. A rule like that is built on a failure the best models have mostly stopped making. As fabricated references get rarer, they become a weaker signal that a text is machine-written, while a human who never checks a real citation can still be wrong.
Benchmarks have the same problem. A test designed around a GPT-3.5 failure becomes uninformative when nearly every newer model passes it, while a test aimed at a persistent tendency such as sycophancy or self-preference stays useful for generations. For developers, the practical rule is to rerun your assumptions whenever the model changes, and to date every claim with the model, version, and evaluation date. The authors used coding agents to speed up the 50 reproductions, which points toward a future where a finding ships with a continuously updated record of which models still show it.
What the study does not show
This is 50 hand-picked claims, not the whole literature, so it does not mean 44% of LLM research is obsolete. The verdicts are fitted trends with wide uncertainty, they depend on the chosen metrics and model panel, and the project has not been peer reviewed. The authors say plainly that the 16-month average is not an expected life span for all findings. A finding can be right when it is published and still stop describing the models people actually use.
Sources
- Owen Queen and James Zou, “The shelf life of LLM findings” (Stanford, November 2, 2026), with code and data
- Owen Queen’s announcement thread on X
- Findings revisited: Walters and Wilder on fabricated citations; Fu et al. on counting letters; Scheurer et al. on insider trading; Sharma et al. on sycophancy; Liu et al., “Lost in the Middle”; Wu et al., “Reasoning or Reciting?”; Huang et al. on self-correction; Lynch et al., “Agentic Misalignment”; Laurito et al. on AI-AI bias; Hofmann et al. on dialect prejudice; Panickssery et al. on self-preference; Dubois et al. on length bias; Press et al. on the compositionality gap; Berglund et al. on the reversal curse; Perez and Ribeiro on prompt hijacking
- ICLR 2026 response to LLM-generated papers and reviews and ICML 2026 peer-review ethics
- Andrew White’s reply on X