Study unveils AI-driven, real-time, hand-object pose estimation framework

A new AI-powered framework has been developed, offering new capabilities for the real-time analysis of two hands engaged in manipulating an object.

A research team led by Professor Seungryul Baek from the UNIST Artificial Intelligence Graduate School has introduced the Query-Optimized Real-Time Transformer (QORT-Former) framework, which accurately estimates the 3D poses of two hands and an object in real time.

The work was published on the arXiv preprint server and was presented at the Annual AAAI Conference on Artificial Intelligence (AAAI), Pennsylvania, USA.

Unlike previous methods that require substantial computational resources, QORT-Former achieves exceptional efficiency while maintaining state-of-the-art accuracy.

To optimize performance, the team proposed a novel query division strategy that enhances query features by leveraging contact information between the hands and the object, in conjunction with a three-step feature update within the transformer decoder. With only 108 queries and a single decoder, QORT-Former achieves 53.5 frames per second (FPS) on an RTX 3090 Ti GPU, making it the fastest known model for hand-object pose estimation.

Professor Seungryul Baek stated, “QORT-Former represents a significant advancement in the understanding of hand-object interactions. It not only enables real-time applications in augmented reality (AR), virtual reality (VR), and robotics, but also pushes the boundaries of real-time AI models.”

“Our work demonstrates that efficiency and accuracy can be optimized simultaneously,” co-first author Khalequzzaman Sayem remarked. “We anticipate broader adoption of our method in fields that require real-time hand-object interaction analysis.”