In context
In the early 2020s, deep-space exploration placed growing demands on autonomous surface rovers to perceive unknown, unstructured extraterrestrial terrain. Semantic segmentation, a core computer-vision task, was typically performed on monocular RGB images, which struggled in complex scenes with similar colors and textures. The emergence of affordable RGB-D sensors offered richer geometric cues, but fusing depth and color efficiently under limited onboard computing remained an open challenge for industrial and field robotics.
What was reported
Researchers at Zhejiang Lab proposed NMRNet, a semantic segmentation network that fuses RGB and depth data using a medium-term fusion framework. The architecture uses parallel ResNet encoders for each modality, followed by refinement-fusion modules that combine multi-scale features, and a decoder with residual refinement blocks to merge cross-level information.
A key contribution is a neighborhood metric relations (NMR) module, which constructs positive and negative sample pairs from the decoder’s feature maps and semantic labels without adding inference overhead. The module leverages local neighborhood context and global class centers to improve feature discrimination, addressing limitations of prior metric-learning methods that required costly sample mining or extra parameters.
Experiments on the NYUDv2 indoor dataset and the MARSv1 Mars simulation site dataset showed that both the RGB-D fusion and the NMR module significantly improved segmentation accuracy compared with existing RGB-D methods, while maintaining inference efficiency.
Why it mattered
By demonstrating that depth information and a lightweight metric-learning module can boost semantic segmentation without increasing runtime cost, the work pointed toward more reliable terrain perception for autonomous robots in extreme environments, a capability directly transferable to industrial inspection and outdoor automation where computational budgets are tight.
"The multi-modal RGB-D information and the neighborhood metric relations can significantly improve the accuracy of semantic segmentation."
Source: 《机器人》期刊 (robot.sia.cn) · Published 2023-03-11 · “一种基于邻域度量关系的RGB-D融合语义分割算法”