When Small Models Beat Giants
Here’s a result that should make anyone rethinking the “bigger is always better” mantra in AI: a 7-billion parameter model, after being taught to reflect on its mistakes, can outperform a 72-billion parameter model that’s ten times its size. This isn’t about finding some obscure edge case—it’s about fundamental improvements in how language models handle complex, verifiable tasks.
The secret sauce? Teaching models to become better at debugging themselves.
The Self-Reflection Problem
Large language models have an odd blind spot. They can memorize vast amounts of information and follow complex instructions, yet they often fail at tasks they should theoretically handle—like calling the right API function or solving basic math equations with given constraints. Even more frustrating, when these models fail, they typically can’t figure out why.
Traditional approaches to this problem involve either training on more data (expensive and often unavailable) or using larger models as teachers (also expensive, and assumes the larger models can actually solve the task). But what if the model could learn to teach itself?
That’s exactly what researchers at Writer, Inc. explored in their recent paper “Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning.” Their approach is elegantly simple: instead of teaching models to be better at specific tasks, teach them to be better at analyzing their own mistakes.
How Reflect, Retry, Reward Works
The methodology operates in three distinct phases, all happening during training:
Reflect: When a model fails a task, it’s prompted to write a short analysis of what went wrong. This isn’t just therapeutic—it’s the foundation for improvement.
Retry: Armed with its self-reflection, the model attempts the same task again. The reflection stays in the conversation context, guiding the second attempt.
Reward: Here’s the clever part. If the second attempt succeeds, only the reflection tokens get rewarded during training. The actual correct answer receives no reinforcement. This forces the model to optimize specifically for generating useful self-analysis, not just memorizing solutions.
The training uses Group Relative Policy Optimization (GRPO), a reinforcement learning technique that works well when you only have binary success/failure feedback. Crucially, this approach is task-agnostic—the model learns general debugging skills rather than specific domain knowledge.
The Numbers Tell a Compelling Story
The researchers tested their approach on two challenging datasets: APIGen (function calling with 60,000 examples) and Countdown (math equation generation with 450,000 problems). The results consistently show smaller trained models punching above their weight class.
Function Calling Results
On the APIGen dataset, the improvements were substantial across all model sizes:
- Qwen-2-1.5B: Jumped from 32.6% to 48.6% accuracy on first attempts (49% improvement)
- Qwen-2-7B: Improved from 66.4% to 72.2% on first attempts, but more impressively, reached 77.3% with reflection—outperforming the vanilla 72-billion parameter Qwen-2-72B model at 76.6%
- Llama-3.1-8B: Gained 3.8 percentage points on first attempts, reaching 74.9% with reflection
Math Equation Performance
The Countdown dataset showed even more dramatic improvements:
- Qwen-2.5-1.5B: Skyrocketed from 6.0% to 34.9% accuracy on first attempts (a 482% relative improvement)
- Qwen-2.5-7B: Improved from 31.7% to 41.6% on first attempts, reaching 50.3% with reflection
- Llama models: Showed the most dramatic gains, with Llama-3.1-8B going from a dismal 2.2% to 8.8% accuracy
The pattern is consistent: models trained with self-reflection not only perform better when they can reflect, but they also improve significantly on their first attempts. This suggests the reflection training enhances general reasoning capabilities.
Why Self-Reflection Actually Works
The magic lies in transforming sparse feedback into dense learning signals. Normally, when a model fails a task, the gradient has to flow backwards through every token in the response, but only the final outcome is labeled as wrong. This creates noisy credit assignment—the model struggles to identify which specific choices led to failure.
Self-reflection changes this dynamic by creating an intermediate step where the model explicitly identifies what went wrong. During training, each reflection token gets its own “was this helpful?” score based on whether it led to success on the retry. This provides much clearer guidance than a simple pass/fail signal.
The researchers found that trained models generate notably different reflections than vanilla models. Before training, reflections tend to be verbose, repetitive, and generic. After training, they become concise, specific, and actionable—more like the debugging notes an experienced programmer would write.
The Anti-Catastrophic Forgetting Effect
One persistent worry with fine-tuning is catastrophic forgetting—models becoming specialized at the expense of general capabilities. The researchers tested their trained models on standard benchmarks (MMLU-Pro, GSM8K, HellaSwag, and MATH) and found minimal degradation. Performance typically remained within 1% of the baseline, with some models even improving slightly.
This stability makes sense given the task-agnostic nature of the training. The models aren’t learning domain-specific tricks; they’re developing general metacognitive skills that transfer broadly.
Limitations and Practical Considerations
The approach isn’t universally applicable. It requires tasks where success can be automatically verified—think API calls that either work or don’t, code that either runs or crashes, or equations that either evaluate correctly or don’t. Tasks requiring subjective judgment or lacking clear success criteria won’t benefit from this method.
The researchers also found that models need some baseline competency to benefit from reflection training. Very small models (like 0.5B parameter variants) simply lacked the capacity to reflect meaningfully or learn from their reflections.
Training efficiency varies significantly between models and tasks. Some experiments converged in just 100 training steps using fewer than 2,000 unique queries, while others required up to 1,750 steps. The method works best when starting from a reasonable baseline rather than trying to teach completely new capabilities.
What This Means for AI Development
This research challenges the prevailing assumption that model size is the primary lever for improving AI capabilities. While bigger models certainly have their place, this work demonstrates that smarter training methods can extract significantly more value from smaller, more efficient models.
The implications extend beyond just cost savings. Self-reflection training could make AI systems more reliable in production environments where consistent performance matters more than peak capability. A 7B model that rarely fails might be more valuable than a 70B model that occasionally produces spectacular results but also spectacular failures.
For practitioners, this suggests a new optimization strategy: instead of always reaching for the largest available model, consider whether a smaller model trained with reflection could meet your needs more efficiently. The 10x size advantage of larger models can often be overcome by better training techniques.
The Bigger Picture
The Reflect, Retry, Reward paradigm represents a shift toward more psychologically plausible AI training. Humans don’t typically learn by processing massive datasets—we learn by making mistakes, reflecting on them, and trying again. This research suggests that AI systems might benefit from similar metacognitive approaches.
Looking ahead, the most interesting question isn’t whether this technique works—the results clearly demonstrate that it does. The question is how far this approach can scale and what other cognitive processes might benefit from similar training methods. If models can learn to debug themselves, what else might they learn to teach themselves?
The path to more capable AI might not always require building bigger models. Sometimes, it requires building smarter ones.
The paper “Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning” was submitted to arXiv on May 30, 2025, by researchers at Writer, Inc.