CONCORD: Asynchronous Sparse Aggregation for Device-Cloud RAG under Document Isolation

Computer Science > Artificial Intelligence [Submitted on 13 Jun 2026] CONCORD: Asynchronous Sparse Aggregation for Device-Cloud RAG under Document Isolation Xuedong Hu, Zhiqing Tang, Zhi Yao, Tian Wang, Weijia Jia Retrieval-augmented generation (RAG) has emerged as a pivotal technique for improving language models by incorporating external knowledge at inference time. As device-cloud collaborative inference makes it feasible to deploy small language models on edge devices, a new setting arises in which private documents remain on the device and public knowledge resides in the cloud. Privacy and policy constraints often forbid raw document exchange, creating a document-isolated dual-end RAG setting. However, existing methods rely on frequent remote synchronization and dense evidence transfer, limiting throughput under realistic latency and bandwidth conditions. To address this issue, we propose CONCORD, an asynchronous sparse aggregation framework for dual-end RAG under document isolation. CONCORD treats the cloud as an asynchronously arriving evidence source rather than a continuously synchronized co-generator. Specifically, we introduce waiting debt control to decide whether each decoding step should continue waiting for remote participation based on the observed return of waiting. We also design a certificate-guided minimal supplementation mechanism that requests only the remote evidence needed to determine the current greedy decision. Steps that consult the cloud preserve the same greedy token as dense dual-end aggregation, while the remaining steps commit locally without remote evidence. Experiments on Natural Questions and WikiText-2 show that CONCORD improves end-to-end throughput over baselines by 1.66\times and 2.15\times, respectively, while reducing per-token communication by over two orders of magnitude and maintaining comparable answer quality and perplexity. Comments: to be published in IEEE ICWS 2026 Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2606.15179 [cs.AI]   (or arXiv:2606.15179v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2606.15179 Focus to learn more Submission history From: Zhiqing Tang [view email] [v1] Sat, 13 Jun 2026 08:00:16 UTC (471 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-06 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)