Mixed Reality Human Robot Interaction Interface using Hand Tracking and Stereo Vision for Unknown and Hazardous Environments

Abstract

Many industries employ human workers to perform hazardous and unsafe tasks that result in injuries and fatalities. These include on-site monitoring of radioactive environments, collisions with heavy machinery on construction sites, and repetitive, physically demanding work. Robotics solutions have been proven to improve task efficiency, product quality, and workplace safety in many industries, though it has posed a challenge to be integrated into dynamic and unstructured environments. This research aims to develop a generalized immersive robot arm teleoperation interface designed to accelerate the deployment of robotics in real-world applications through an intuitive Human-Robot Interaction (HRI) system. The methodology involves mounting a ZEDM stereo camera, attached to a 2-axis servo-actuated gimbal, on the end effector of a 6-axis robot arm. Stereo images are streamed to a Meta Quest 2 virtual reality (VR) head-mounted display (HMD), providing a first-person view (FPV) with human-like perception of the end effector's surroundings. Hand tracking is used to map the operator’s hands into the FPV, enabling omnidirectional motion control and gripper operation via gesture recognition. In-situ data is spatially mapped to the stereoscopic view, creating a mixed-reality (MR) HRI interface. Results show improved depth perception and higher control stability compared to traditional monocular video systems paired with keyboard controls. Specifically, the MR interface achieves a 39% reduction in task completion time during target-tracking experiments and demonstrates five times greater stability in trajectory control tests. This MR HRI interface promotes the safe use of robotics in unknown or hazardous environments, allowing operators to perform high-risk tasks remotely using only HMDs and natural hand gestures.