On Divergence Measures for Training GFlowNets

Computer Science > Machine Learning [Submitted on 12 Oct 2024 (v1), last revised 21 Oct 2024 (this version, v2)] On Divergence Measures for Training GFlowNets Tiago da Silva, Eliezer de Souza da Silva, Diego Mesquita Generative Flow Networks (GFlowNets) are amortized inference models designed to sample from unnormalized distributions over composable objects, with applications in generative modeling for tasks in fields such as causal discovery, NLP, and drug discovery. Traditionally, the training procedure for GFlowNets seeks to minimize the expected log-squared difference between a proposal (forward policy) and a target (backward policy) distribution, which enforces certain flow-matching conditions. While this training procedure is closely related to variational inference (VI), directly attempting standard Kullback-Leibler (KL) divergence minimization can lead to proven biased and potentially high-variance estimators. Therefore, we first review four divergence measures, namely, Renyi-\alpha's, Tsallis-\alpha's, reverse and forward KL's, and design statistically efficient estimators for their stochastic gradients in the context of training GFlowNets. Then, we verify that properly minimizing these divergences yields a provably correct and empirically effective training scheme, often leading to significantly faster convergence than previously proposed optimization. To achieve this, we design control variates based on the REINFORCE leave-one-out and score-matching estimators to reduce the variance of the learning objectives' gradients. Our work contributes by narrowing the gap between GFlowNets training and generalized variational approximations, paving the way for algorithmic ideas informed by the divergence minimization viewpoint. Comments: Accepted at NeurIPS 2024, this https URL Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) MSC classes: 68T05 ACM classes: G.3; I.5.1; I.2.8; I.2.6 Cite as: arXiv:2410.09355 [cs.LG]   (or arXiv:2410.09355v2 [cs.LG] for this version)   https://doi.org/10.48550/arXiv.2410.09355 Focus to learn more Submission history From: Eliezer De Souza Da Silva [view email] [v1] Sat, 12 Oct 2024 03:46:52 UTC (4,010 KB) [v2] Mon, 21 Oct 2024 20:27:30 UTC (4,010 KB) Access Paper: HTML (experimental) view license Current browse context: cs.LG < prev   |   next > new | recent | 2024-10 Change to browse by: cs cs.AI stat stat.ML References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)