Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces

Computer Science > Artificial Intelligence [Submitted on 3 Jun 2026] Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces Nicolás Astorga, Nabeel Seedat, Mihaela van der Schaar Verifiable reward training has improved mathematical and coding reasoning, but these domains capture only part of step-by-step decision making. Many real-world tasks require finding a high-value feasible plan among many valid alternatives. We introduce OPT*, a scalable family of optimization-style tasks for training and evaluating LLM step-by-step optimization-like reasoning along a complexity axis: each task provides a feasibility checker and evaluator, while a complexity parameter expands the search space without requiring new human labels. This motivates studying these tasks in two regimes: (i) solver-guided online policy optimization, which uses a solver as a value oracle for partial states and applies rank-based reward shaping to reinforce better next steps, and (ii) search-based offline RL when such solvers are unavailable. Theoretically, we relate success in large search spaces to the information a reasoner extracts per unit of search budget. Empirically, we ablate the ingredients that make search efficient on OPT* and show that training on OPT* improves step-by-step optimization-like reasoning. Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2606.05464 [cs.AI]   (or arXiv:2606.05464v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2606.05464 Focus to learn more Submission history From: Nicolas Astorga [view email] [v1] Wed, 3 Jun 2026 21:43:38 UTC (1,089 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-06 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)