Object detection using coaxial images

For an autonomous mobile system such as an autonomous robot that moves throughout a city, semantic segmentation is important. Performing semantic segmentation under diverse conditions, in turn, requires 1) a robust ability to recognize objects in low-visibility environments, such as at night and 2) the ability to recognize objects that transmit visible light, such as glass and acrylic used in doors and windows. To satisfy these requirements, using RGB images and infrared images simultaneously is considered effective. Visibility and infrared transmission characteristics are different for different objects; therefore, merely entering them into the conventional semantic segmentation framework is not applicable. For example, when a pedestrian is present behind a glass, the visible image captures the pedestrian rather than the glass and the infrared image captures the glass. In this research, we propose a new semantic segmentation method having a three-stream structure, focusing on the difference in the transmission characteristics. This method extracts not only valid features for ordinary non-transparent objects but also features effective for the recognition of transparent objects by utilizing differences in objects to be imaged owing to transmission characteristics. Furthermore, we constructed a new dataset called “coaxials” for the visible and infrared coaxial dataset and demonstrated that we can obtain better segmentation performance compared with the conventional method.

Publications

1. Tomoyuki Takahata and Tatsuya Harada, “Glass detection using visible and long-wavelength infrared coaxial camera,” The 40th annual conference of the Robotics Society of Japan (RSJ2022), 2B1-03, 2022. (Japanese domestic conference)
2. Atsuro Okazawa, Tomoyuki Takahata, and Tatsuya Harada, “Simultaneous transparent and non-transparent objects segmentation with multispectral scenes,” The 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2019), Macao, China, 4–8 November, pp. 4977–4984, 2019. [Proceedings]
3. Coaxials dataset is available on github.