Generalized Few-Shot Semantic Segmentation via Contrastive Learning and Orthogonal Decoupling

Contribution. Propose CLOD-GFSS that integrates multi-scale contextual contrastive learning and hyperspherical orthogonal decoupling to address base-class feature drift and novel-class generalization in GFSS

Overview

CLOD-GFSS addresses two key limitations of existing GFSS methods: base-class feature degradation during novel-class fine-tuning, and intra-class variations among limited novel-class samples. The method proposes a hierarchical context-aware architecture (CIAM) for multi-scale contrastive learning, and an orthogonal decoupling module (ODM) that constrains feature space geometry via hyperspherical pseudo-target matching.

Research Problem

Existing GFSS two-stage training paradigms cause novel-class fine-tuning to disrupt base-class representations. Prototype-based methods struggle with significant intra-class variations from limited annotated novel-class samples. These issues cause performance degradation on both base and novel classes.

Core Idea

Combine contrastive learning with orthogonal decoupling: (1) Multi-scale anchor representations fuse global semantics with local details, optimized via Pixel-Anchor contrastive loss for intra-class compactness and inter-class separation; (2) Hyperspherical orthogonal decoupling generates uniformly distributed pseudo-targets and uses Hungarian algorithm for dynamic anchor-target matching to suppress feature drift.

Key Components

CIAM

Contextual Information Awareness Module

A multi-scale context fusion mechanism that progressively integrates global semantic features with local detail features through weighted anchor updates. Pixel-Anchor (PA) contrastive loss enforces intra-class feature aggregation around class-specific anchors and inter-class separation across all encoding layers.

ODM

Orthogonal Decoupling Module

Generates uniformly distributed pseudo-target vectors on a d-dimensional hypersphere via Target Generation Loss, then uses Hungarian algorithm for online optimal matching between evolving category anchors and pseudo-targets. Pseudo-Target Matching Contrastive (PTMC) loss enforces feature space topology preservation.

Two-Phase Training

Dual-Phase Progressive Training Strategy

Phase 1: fully supervised base-class learning with CIAM and ODM for discriminative feature space. Phase 2: novel-class update with frozen feature extractor to minimize base-class interference, while CIAM and ODM enhance novel-class feature discriminability through the new classifier.

The overall framework employs a two-phase training strategy: Phase 1 trains the full model (encoder + CIAM + ODM + classifier) on base classes with CE + PA + OrCo losses. Phase 2 freezes the encoder and fine-tunes CIAM, ODM, and a new classifier for novel classes, minimizing interference with base-class representations.

CIAM computes category-specific anchor points at each encoding layer by averaging feature vectors per class, then fuses local anchors with global anchors via weighted combination (optimal ratio 0.7:0.3). The Pixel-Anchor contrastive loss optimizes feature embeddings by pulling intra-class features closer to their anchors while pushing inter-class features apart.

ODM generates mutually orthogonal pseudo-targets on a hypersphere via Target Generation Loss, then uses Hungarian algorithm for online optimal matching between category anchors and pseudo-targets. This prevents static random assignment from disrupting semantic relationships, achieving +1.2% Total mIoU improvement over random assignment.

Results

PASCAL-5i (5-shot)

71.86% Total mIoU

+1.58% over POP (70.28%), SOTA

Best overall mIoU on PASCAL-5i 5-shot

PASCAL-5i (1-shot)

39.22% Novel mIoU

+3.71% over POP (35.51%), best novel-class

Significant improvement in novel-class generalization

COCO-20i (5-shot)

49.37% Total mIoU

+0.62% over POP (48.75%), SOTA

Best overall mIoU on COCO-20i 5-shot

COCO-20i (5-shot)

31.15% Novel mIoU

+0.55% over POP (29.97%), SOTA

Best novel-class mIoU on COCO-20i 5-shot

CLOD-GFSS achieves state-of-the-art on both benchmarks. On PASCAL-5i 1-shot, novel-class mIoU improves by +3.71% over POP. Ablation shows CIAM adds +0.74% Total mIoU, ODM further adds +3.82% Novel mIoU. Online matching outperforms random assignment by +1.2% Total mIoU. The method reduces boundary errors by 35.7% and improves small-object recall by 28.4% compared to POP.

Conclusion

CLOD-GFSS integrates contrastive learning and orthogonal decoupling for GFSS. The CIAM module enhances multi-scale feature discrimination, while the ODM module suppresses base-class feature drift through hyperspherical orthogonal constraints. Experiments demonstrate consistent improvements on PASCAL-5i and COCO-20i, with notable gains in novel-class generalization and boundary precision.

Takeaway. Combining multi-scale contrastive learning with orthogonal feature space decoupling effectively addresses the dual challenge of base-class preservation and novel-class generalization in GFSS. The key insight is that constraining feature space geometry via orthogonal pseudo-targets on a hypersphere preserves base-class representations while enhancing novel-class discriminability.

BibTeX

@article{jiang2026clod,
  title={Generalized few-shot semantic segmentation via contrastive learning and orthogonal decoupling},
  author={Jiang, Lulu and Xia, Yaozheng and Wang, Shaorong},
  journal={Multimedia Systems},
  volume={32},
  number={171},
  publisher={Springer},
  year={2026},
  doi={10.1007/s00530-026-02227-8}
}