Federated learning enhanced multi-modal sensing and perception in a collaborative multi-robot system

Abstract

Multi-robot systems are increasingly essential across a wide array of sectors, such as industrial automation, transportation, and search and rescue. The key to these systems lies in the capabilities of agents to collaboratively perceive, comprehend, and reason about their surroundings, thereby attaining advanced situational awareness. Recent advances in artificial intelligence, especially in the field of deep learning (DL), have increased the ability of multi-robot systems to effectively utilize and understand data produced by various sensors. Despite numerous efforts to integrate multiple sensors, this area remains complex and challenging due to heterogeneous, unstructured, and cluttered deployment environments. Furthermore, these operating scenarios vary considerably across different settings, including hospitals, private residences, ports, and other contexts where privacy and security prevail.

This dissertation addresses these challenges by integrating multi-modal sensors to enhance high-level robot perception across multiple agents while ensuring security and privacy through Federated Learning (FL). FL, a privacy-preserving DL method, distributes learning across isolated data silos, enabling secure knowledge sharing among robots via model transfers instead of direct data exchanges.

The research begins by investigating the limitations of existing multi-modal sensor datasets and employing diverse sensors, including LiDAR (spinning and solid-state LiDARs), visual sensors, Inertial Measurement Units (IMUs), and UltraWideband (UWB), to create more comprehensive datasets. After benchmarking current state-of-the-art SLAM and LiDAR odometry (LO) algorithms, the study develops novel multi-robot relative localization approaches as a foundation for other perception tasks. It then explores using LiDAR-generated images and solid-state LiDAR to enhance UAV tracking and LO. Finally, the effectiveness of FL is demonstrated through a case study on multi-robot visual obstacle avoidance (VOA), transitioning from simulation to real-world scenarios. By incorporating LiDAR and cameras in real-world applications, the research achieves lifelong learning on VOA within the FL framework. This case study highlights FL’s practical applications and advantages, suggesting its potential generalizability across a broad range of robotic perception tasks.