Meta developers have introduced an innovative machine vision system for robots

Meta Reality Labs has introduced a new HOT3D dataset that could significantly impact the development of computer vision and robotics technologies. The dataset contains over 833 minutes of egocentric 3D videos captured using Project Aria glasses and the Quest 3 helmet. The recordings involved 19 people interacting with 33 different objects in everyday and office environments.

The dataset contains approximately 3.7 million images, which are accompanied by detailed annotations. Among them — 3D poses of objects, hands, cameras, and 3D models of hands and objects. These data allow us to study complex tasks such as 3D hand tracking, object pose estimation, and object motion modeling in hands.

Experiments have shown the high efficiency of using HOT3D. The multi-camera approach used in the dataset significantly outperforms single-camera systems in solving tasks related to motion and object position recognition.

HOT3D is open to researchers around the world. It can become the basis for the development of human-machine interfaces, augmented and virtual reality systems, and for improving the interaction of robots with the environment.