The Hidden Power of Scaling Factor in LoRA Optimization

Computer Science > Artificial Intelligence [Submitted on 11 Jun 2026] The Hidden Power of Scaling Factor in LoRA Optimization Zicheng Zhang, Haoran Li, Jiaxing Wang, Guoqiang Gong, Anqi Li, Yudong Hu, Ting Xiong, Yurong Gao, Junxing Hu, Zhida Jiang, Yifeng Zhang, Pengzhang Liu, Qixia Jiang In Low-Rank Adaptation (LoRA), the scaling factor \alpha is often treated as a mere complement to the learning rate, yet its role in optimization remains poorly understood. In this paper, we reveal that the scaling factor \alpha and the learning rate function differently, with \alpha emerging as the dominant driver of effective optimization, delivering gains that cannot be replicated by learning rate scaling alone. Through the synergy of extensive empirical analysis and a theoretical Signal-Drift framework, we uncover three findings into LoRA's scaling mechanism: First, LoRA's spectral suppression smooths the optimization landscape, rendering standard hyperparameters overly conservative and creating an optimization gap. Second, when leveraging this smoothness to accelerate convergence, \alpha outperforms the learning rate by amplifying the task signal without increasing the drift ratio. Third, the optimal scaling factor follows a sublinear relationship with the rank, well characterized by a square-root law with an unexpectedly large coefficient, revealing the insufficient scaling of existing rank-tied heuristics. Based on these insights, we propose LoRA-\alpha, a minimalist framework that restores \alpha to its principled regime, making LoRA compatible with standard small learning rates. Extensive evaluations across diverse tasks demonstrate that LoRA-\alpha consistently improves performance while streamlining hyperparameter search, unleashing the learning potential of LoRA. Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2606.12883 [cs.AI]   (or arXiv:2606.12883v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2606.12883 Focus to learn more Submission history From: Zicheng Zhang [view email] [v1] Thu, 11 Jun 2026 04:19:32 UTC (1,463 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-06 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)