GAPFormer: Geometry-Adaptive Propagated Transformer for Point Cloud Representation

Contribution. Propose a geometry-adaptive Transformer for 3D point cloud semantic segmentation that integrates spherical harmonic geometric encoding, dynamic positional bias, and point-wise convolution attention

Overview

GAPFormer addresses the limited geometric modeling capabilities of Transformer-based methods in 3D point cloud segmentation. It proposes three key modules: SHGFM for direction-aware geometric encoding via spherical harmonics, Geo-DPB for geometry-aware dynamic positional bias, and PCBA for multi-level channel-spatial feature fusion. The framework achieves state-of-the-art performance on ScanNet v2, S3DIS, and ScanNet200 benchmarks.

Research Problem

Existing Transformer-based point cloud methods lack sufficient geometric awareness - they struggle with representing complex local structures and fail to utilize geometric prior knowledge effectively. Conventional coordinate embedding through MLPs yields insufficient encoding of 3D spatial relationships.

Core Idea

Enhance Transformer-based point cloud segmentation by explicitly injecting geometric priors at multiple levels: (1) spherical harmonic basis functions for direction-aware feature encoding, (2) geometry-guided dynamic positional bias for attention enhancement, and (3) point-wise convolution-based channel-spatial co-optimization for feature refinement.

Key Components

SHGFM

Spherical Harmonic Geometric Feature Modulator

Leverages learnable spherical harmonic basis functions to encode direction-aware geometric relationships in local neighborhoods. By computing polar/azimuthal angles and low-order real-valued SH coefficients, it captures anisotropic geometric structures that conventional coordinate embedding misses.

Geo-DPB

Geometrically-Aware Dynamic Positional Bias

A three-stage progressive module that precomputes 7D geometric feature vectors (offsets, distance, direction), maps them through a residual MLP to generate dynamic attention biases stored in a hash dictionary, and injects these biases into the standard attention mechanism.

PCBA

Point-Wise Convolution-Based Attention

Refines the CBAM channel-spatial attention mechanism for 3D point clouds. Uses average/max pooling for channel attention and Conv1D for spatial attention, achieving simultaneous optimization of channel-wise and spatial feature representations while maintaining computational efficiency.

The overall architecture follows an encoder-decoder paradigm: raw point clouds are transformed into high-dimensional features via sparse convolution, then a hierarchical Transformer encoder progressively learns multi-scale geometric features with cascaded pooling, while the decoder upsamples to reconstruct fine spatial details for accurate segmentation.

SHGFM computes direction vectors, magnitudes, and normalized vectors for each neighbor, then derives polar/azimuthal angles to calculate spherical harmonic basis functions (degree 0-2). These SH coefficients are concatenated with original features and relative distances, then fused through a residual MLP to produce geometry-prior-enhanced Key-Value pairs.

Geo-DPB precomputes all possible 7D geometric feature vectors within a local 3D neighborhood, maps them through a 3-layer MLP (ReLU+GeLU activations) to generate positional biases stored in a hash dictionary for efficient inference. During attention, biases are retrieved via quantized offsets and additively injected into attention weights.

Results

ScanNet v2 (Val)

77.35% mIoU

+0.23% over PTv3 (77.12%)

State-of-the-art on ScanNet v2 validation set

S3DIS (Area 6)

72.6% mIoU

+0.8% over PTv3 (71.8%)

Significant improvement on S3DIS with OA=91.8%

ScanNet200 (Test)

36.06% mIoU

+0.86% over PTv3 (35.2%)

Superior performance on fine-grained 200-class segmentation

S3DIS (Area 6)

91.8% OA

+0.4% over PTv3 (91.4%)

Highest overall accuracy on S3DIS

GAPFormer achieves consistent improvements across all three benchmarks. On S3DIS, it notably improves per-class mIoU for board (+5.4%), bookcase (+2.9%), and sofa (+2.1%) over PTv3, demonstrating superior fine-grained structure segmentation. Ablation study confirms that all three modules contribute synergistically, with the full model achieving the best mIoU of 72.6%.

Conclusion

GAPFormer introduces three geometry-aware innovations to address the limited geometric modeling capabilities of Transformer-based methods. The synergistic combination of SHGFM, Geo-DPB, and PCBA achieves state-of-the-art performance across ScanNet v2, S3DIS, and ScanNet200, with notable gains in fine-grained object segmentation. The transferable design offers potential value for broader 3D vision tasks.

Takeaway. Explicitly encoding geometric priors through spherical harmonics and dynamic positional biases significantly enhances Transformer-based point cloud segmentation. The key insight is that direction-aware frequency-domain encoding (SHGFM) combined with geometry-guided attention bias (Geo-DPB) addresses the fundamental limitation of conventional coordinate embedding approaches.

BibTeX

@inproceedings{zheng2025gapformer,
  title={GAPFormer: Geometry-Adaptive Propagated Transformer for Point Cloud Representation},
  author={Zheng, Wanlu and Wang, Shaorong},
  booktitle={Pattern Recognition and Computer Vision (PRCV)},
  pages={336--350},
  year={2025},
  publisher={Springer},
  doi={10.1007/978-981-95-5737-0_24}
}