CARGO-VL: Counterfactual Arbitration with Risk-Constrained Group Optimization for Vision-Language Models

Computer Science > Artificial Intelligence [Submitted on 5 Aug 2026] CARGO-VL: Counterfactual Arbitration with Risk-Constrained Group Optimization for Vision-Language Models De Jiang, Zhengyang Zhang, Kehong Yuan, Shaohua Ma Vision-language systems combine images with retrieved text, but these sources can disagree or jointly fail to support an answer. Reliable models must identify the trustworthy source and abstain when neither is adequate. Existing post-training objectives score instances independently and therefore do not enforce coherent behavior under counterfactual evidence changes. We introduce CARGO-VL, a group-relative framework that optimizes matched variants covering aligned, image-correct, text-correct, and both-wrong (A/V/T/N) evidence states as one bundle. Its objective couples condition-wise correctness with transition rewards for answer invariance, source equivariance, and answer-to-abstention switching, while a primal-dual controller balances unsafe answers against excessive deferral. We also contribute XMC (eXtended Modal Conflict), a four-condition conflict training resource, and evaluate transfer on CMC-Bench and Modality-Bias. Across multiple seeds, CARGO-VL improves conflict handling, unsupported-answer avoidance, and modality balance over pointwise baselines. Ablations identify complementary benefits from relational transition signals and adaptive risk control, supporting counterfactual consistency as a practical objective for reliable multimodal evidence arbitration. Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2608.04509 [cs.AI]   (or arXiv:2608.04509v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2608.04509 Focus to learn more Submission history From: De Jiang [view email] [v1] Wed, 5 Aug 2026 06:47:46 UTC (4,735 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-08 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)