Asymmetric Dual-Stream Networks for Lightweight RGB-D Salient Object Detection

Contribution. Proposes an asymmetric dual-stream lightweight RGB-D salient object detection network with only 5.1M parameters and 0.77G FLOPs, achieving superior performance across five benchmark datasets.

Overview

This paper proposes an asymmetric dual-stream lightweight RGB-D salient object detection network. By designing lightweight feature extraction networks of different capacities for each modality, the parameter count is significantly reduced. A Multi-modal Feature Enhancement Fusion (MFEF) module compensates for information loss in the lightweight backbone, while a Global Context Module (GCM) enables dense decoding. Experiments on five benchmark datasets demonstrate that the network outperforms most mainstream models in both quantitative and qualitative metrics, with only 5.1M parameters and 0.77G FLOPs.

Research Problem

RGB-D salient object detection models suffer from increased parameters and computational costs due to multi-modal inputs, making deployment on resource-constrained devices difficult. Existing lightweight methods use symmetric networks for both modalities, ignoring the fundamental differences between RGB and depth images and causing parameter redundancy.

Core Idea

Based on the observation that RGB images contain richer semantic information than depth images, design backbone networks of different capacities for each modality (asymmetric design), with the depth backbone having only 55% of the RGB backbone parameters. MFEF and GCM modules compensate for information loss caused by lightweight design.

Key Components

Asymmetric Backbone

Asymmetric Feature Extraction

Designs MobileNetV3-based backbones of different capacities for RGB and depth streams, reducing depth backbone to 55% of RGB backbone parameters.

MFEF

Multi-modal Feature Enhancement Fusion

Uses inverted residual blocks with attention to select representative RGB features, compensating for lightweight backbone information loss.

GCM

Global Context Dense Decoding

Employs multi-branch multi-scale feature extraction (1x1, 3x3, 5x5, 7x7 dilated convolutions) with dense connections to propagate high-level semantics.

The encoder-decoder network uses MobileNetV3-Large (last three layers removed) for RGB and a lighter network for depth, each producing five-level features. Features are fused via MFEF modules and decoded through GCM dense connections, with predictions at four levels for deep supervision.

MFEF first multiplies RGB and depth features element-wise, enhances via IRB, then generates RGB attention through GAP+FC+ReLU+FC+sigmoid. The attention, enhanced features, and raw depth features are combined via IRB to produce fused features that compensate for lightweight backbone limitations.

GCM uses dense connections: each decoding sub-module output feeds all subsequent sub-modules. Internally, 1x1 convolution reduces dimensions, then four branches (1x1, 3x3, 5x5, 7x7 dilated convolutions) extract multi-scale features enhanced by channel attention.

Results

Parameters

5.1M Params

59% of MoADNet

Ultra-lightweight

FLOPs

0.77G FLOPs

Lowest

Significantly lower

NJU2K

0.038 MAE

Best

LFSD

0.875 S-measure

+1.86% vs MoADNet

Leading lightweight

Across five benchmark datasets, the network achieves best or second-best performance on most metrics. On LFSD, compared to MoADNet: MAE reduced 9.37%, S-measure +1.86%, F-measure +2.63%, E-measure +1.77%, while using only 5.1M params and 0.77G FLOPs.

Conclusion

The asymmetric dual-stream lightweight network effectively reduces computational costs through asymmetric backbones, compensates for lightweight backbone limitations via MFEF, and enhances global context through GCM dense decoding. Experiments demonstrate the dual advantage of accuracy and lightweight design.

Takeaway. Designing modality-specific backbones of different capacities (asymmetric design) is an effective strategy to reduce parameter redundancy. With well-designed fusion and decoding modules, lightweight networks can match or surpass the performance of complex networks.

BibTeX

@article{wang2024asymmetric,
  title   = {Asymmetric Dual-Stream Networks for Lightweight {RGB-D} Salient Object Detection},
  author  = {Wang, Yan and Zheng, Wanlu and Xia, Yaozheng and Wang, Shaorong},
  journal = {Journal of Shanghai Jiao Tong University (Science)},
  year    = {2024},
  doi     = {10.1007/s12204-024-2794-0},
  url     = {https://link.springer.com/article/10.1007/s12204-024-2794-0},
  issn    = {1995-8188}
}