A new paper from researchers at Google DeepMind demonstrates the potential for large language models (LLMs) like GPT-3 to act as general-purpose optimization algorithms. The approach, called OPRO (Optimization by PROmpting), simply describes an optimization problem to the LLM using natural language prompts. The LLM then iteratively generates new candidate solutions aiming to improve on previous ones.
The researchers tested OPRO on small instances of classic optimization problems like linear regression and the traveling salesman problem. With no specialized training, LLMs were able to find high-quality solutions competitive with standard algorithms.
The most promising results came from optimizing prompts themselves. The goal was to find an instructional prompt that maximizes an LLM’s accuracy on reasoning tasks like mathematical word problems. Across benchmarks like GSM8K and Big-Bench Hard, optimized prompts improved accuracy over human-designed prompts by up to 50%, respectively. The prompts generalized to unseen test data, sometimes matching state-of-the-art few-shot performance.
Overall, the work shows LLMs have an intrinsic capability for iterative optimization just by leveraging patterns from previous solution attempts provided in natural language. With their vast knowledge and few-shot learning abilities, LLMs could prove to be versatile black-box optimizers – adapting to new problems simply by updating the prompt descriptions. If the approach scales to larger instances, it could have widespread implications across fields like machine learning, engineering design, and operations research. Rather than designing custom algorithms, OPRO suggests we may one day simply describe optimization problems to a general-purpose LLM.