Generalized Few-Shot Semantic Segmentation Based on Relevant Intrinsic Feature Enhancement

Contribution. Propose RIFE method that addresses intra-class variations in generalized few-shot semantic segmentation through feature decoupling/recombination and joint learning, achieving state-of-the-art on PASCAL-5i and COCO-20i

Overview

RIFE tackles intra-class variations in generalized few-shot semantic segmentation by proposing a feature decoupling and recombination mechanism that mines robust intrinsic features, and a joint learning strategy that concurrently segments both support and query images during novel class inference to verify prototype discriminability.

Research Problem

Existing prototype-based methods suffer from intra-class variations between support and query images that degrade prototype effectiveness. Current generalized methods also exclude novel-class images during base training and require base-class labels in support annotations, losing valuable data and adding annotation overhead.

Core Idea

Decouple foreground/background features across support-query branches via self-attention and cross-attention mechanisms, recombine them to reduce intra-class variations while enhancing inter-class discrimination, and validate prototypes by jointly segmenting both support and query images during inference.

Key Components

Feature Decoupling

Feature Decoupling and Recombination Mechanism

Separates support features into foreground/background via annotation masks, drives query feature decoupling through cross-branch foreground interaction without query mask supervision, then uses self-attention and cross-attention to enhance intra-branch discrimination and inter-branch semantic alignment

Joint Learning

Joint Support-Query Segmentation Strategy

Concurrently predicts segmentation masks for both support and query images during novel class inference, using support predictions to validate prototype discriminability and jointly optimize both branches for more accurate and stable prototype representations

Ensemble Module

Base-Novel Class Ensemble Fusion

Fuses base-class and novel-class prediction branches using a threshold-based strategy: novel class predictions take priority when confidence exceeds threshold, otherwise base class predictions are used, enabling seamless integration of both segmentation outputs

RIFE comprises three core components: Base Prediction (standard supervised learning on base classes via PSPNet with ResNet-50 backbone), Novel Prediction (meta-learning with feature decoupling, prototype generation, and joint learning), and Ensemble Module (threshold-based fusion of base and novel predictions).

Self-attention within each branch refines foreground features by computing Q/K/V projections and applying scaled dot-product attention with residual connections, enhancing discriminability of task-relevant features in both support and query branches.

Cross-attention between branches aligns support and query foreground features by computing attention weights from support features to query features, reducing intra-class support-query discrepancies while promoting foreground-background decoupling across branches.

Results

PASCAL-5i (5-shot)

71.00% Total mIoU

+0.72% over POP (70.28%), SOTA

Best overall mIoU on PASCAL-5i 5-shot setting

PASCAL-5i (1-shot)

42.65% Novel mIoU

+1.41% over BCM (41.24%), SOTA

Best novel-class mIoU in 1-shot setting

COCO-20i (5-shot)

48.89% Total mIoU

+0.14% over POP (48.75%), SOTA

Best overall mIoU on COCO-20i 5-shot setting

COCO-20i (5-shot)

30.95% Novel mIoU

+0.35% over BCM (30.60%), SOTA

Best novel-class mIoU on COCO-20i 5-shot

RIFE achieves state-of-the-art performance on both PASCAL-5i and COCO-20i benchmarks across 1-shot and 5-shot settings. The feature decoupling and recombination mechanism (method c) outperforms simpler variants (methods a, b) by +3.16% Total mIoU on PASCAL-5i 5-shot. Joint learning strategy (strategy b) adds +1.23% Total mIoU over query-only optimization.

Conclusion

RIFE addresses intra-class variations in generalized few-shot semantic segmentation through feature decoupling/recombination and joint learning. The method eliminates two key constraints in existing methods: exclusion of novel classes during base training and requirement for base-class labels in support annotations. Experiments on PASCAL-5i and COCO-20i demonstrate consistent state-of-the-art performance.

Takeaway. Cross-branch feature decoupling and recombination is an effective strategy for reducing intra-class variations in few-shot segmentation. Joint learning that validates prototypes by concurrently segmenting both support and query images improves prototype discriminability and segmentation accuracy without requiring additional annotations.

BibTeX

@inproceedings{jiang2025generalized,
  title={Generalized Few-Shot Semantic Segmentation Based on Relevant Intrinsic Feature Enhancement},
  author={Jiang, Lulu and Xia, Yaozheng and Wang, Shaorong},
  booktitle={Pattern Recognition and Computer Vision (PRCV)},
  pages={234--247},
  year={2025},
  publisher={Springer},
  doi={10.1007/978-981-95-4987-0_17}
}