Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models

Computer Science > Computer Vision and Pattern Recognition [Submitted on 27 Mar 2026] Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models Zhuan Shi, Alireza Dehghanpour Farashah, Rik de Vries, Golnoosh Farnadi Concept erasure in text-to-image diffusion models seeks to remove undesired concepts while preserving overall generative capability. Localized erasure methods aim to restrict edits to the spatial region occupied by the target concept. However, we observe that suppressing a concept can unintentionally weaken semantically related neighbor concepts, reducing fidelity in fine-grained domains. We propose Neighbor-Aware Localized Concept Erasure (NLCE), a training-free framework designed to better preserve neighboring concepts while removing target concepts. It operates in three stages: (1) a spectrally-weighted embedding modulation that attenuates target concept directions while stabilizing neighbor concept representations, (2) an attention-guided spatial gate that identifies regions exhibiting residual concept activation, and (3) a spatially-gated hard erasure that eliminates remaining traces only where necessary. This neighbor-aware pipeline enables localized concept removal while maintaining the surrounding concept neighborhood structure. Experiments on fine-grained datasets (Oxford Flowers, Stanford Dogs) show that our method effectively removes target concepts while better preserving closely related categories. Additional results on celebrity identity, explicit content and artistic style demonstrate robustness and generalization to broader erasure scenarios. Comments: Accepted by CVPR 2026 main Subjects: Computer Vision and Pattern Recognition (cs.CV); Cryptography and Security (cs.CR) Cite as: arXiv:2603.25994 [cs.CV]   (or arXiv:2603.25994v1 [cs.CV] for this version)   https://doi.org/10.48550/arXiv.2603.25994 Focus to learn more Submission history From: Zhuan Shi [view email] [v1] Fri, 27 Mar 2026 00:58:11 UTC (42,772 KB) Access Paper: HTML (experimental) view license Current browse context: cs.CV < prev   |   next > new | recent | 2026-03 Change to browse by: cs cs.CR References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)