Human-Following System Based on Deep Learning for Supermarket Robot

Abstract

Physical burden and difficulty pushing shopping trolleys have become the issues in supermarket shopping activities, making shopping less convenience. In order to address the issues while promoting customer experience, innovative solutions such as robotics technologies are necessary, eliminating the need to manually push trolleys. Nevertheless, frequently, interaction with robots needs less intuitive methods. Thus, the aim of this research is to design a prototype of supermarket robot. An easy-to-control robot that can follow its user dynamically through hand gestures.  Foxy's Robot Operating System (ROS) 2 framework is used to design the robot system, involving RGB camera, whose data is processed by the YOLOv11-Pose deep learning model for human detection and hand signals, to provide visual perception. The triangulation principle based on the pixel distance between the head and neck keypoints is utilized to calculate the distance, during which the hand gesture interpretation system sets the robot operational status. The results indicate that the F1-score of the object detection model is greater than 0.92, with an inference speed of 14 FPS. While the human distance estimation algorithm recorded high accuracy, it achieved an average Mean Absolute Percentage Error (MAPE) of below 5.3%. The reliable hand gesture detection system is proven by average detection confidence values above 0.83 at distances of 1.5 to 5 meters. On the supermarket simulation track, the robot responded and followed the human movement stably and successfully. Moreover, it can perform several commands using hand signals for huma. It follows stop mode activation and moves forward, backward, and maneuvering.