A new model to plan and control the movements of humanoids in 3D environments

Humanoids, robotic or virtual systems with body structures that resemble the human body, have a wide range of real-world applications. As their limbs and bodies mirror those of humans, they could be made to reproduce a wide range of human movements, such as walking, crouching, jumping, swimming and so on.

Related posts

AI transforms Bitcoin price prediction and market strategy

May 19, 2025

Social robots learning without us? New study cuts humans from early testing

May 19, 2025

Computationally generating realistic motions for virtual humanoid characters could have interesting implications for the development of video games, animated films, virtual reality (VR) experiences, and other media content. Yet the environments portrayed in video games and animations are often highly dynamic and complex, which can make planning motions for humanoids introduced in these environments more challenging.

Researchers at NVIDIA Research in Israel recently introduced PlaMo (Plan and Move), a new computational approach to plan the movements of humanoids in complex, 3D, physically simulated worlds. Their approach, presented in a paper published on arXiv preprint server, consists of a scene-aware path planner and a robust control policy.

“Controlling humanoids in complex physically simulated worlds is a long-standing challenge with numerous applications in gaming, simulation, and visual content creation,” Assaf Hallak, Gal Dalal, and their colleagues wrote in their paper.

“In our setup, given a rich and complex 3D scene, the user provides a list of instructions composed of target locations and locomotion types. To solve this task, we present PlaMo, a scene-aware path planner and a robust physics-based controller.”

Most previous studies aimed at planning the movements of humanoid characters in 3D simulated environments focused on the development of either a planner or a controller rather than both. As the tasks these two types of models perform (i.e., planning and executing a humanoid’s motions) are interdependent, Hallak, Dalal and their colleagues set out to devise a computational approach that tackled them both.

“The path planner produces a sequence of motion paths, considering the various limitations the scene imposes on the motion, such as location, height, and speed,” Hallak, Dalal and their colleagues wrote. “Complementing the planner, our control policy generates rich and realistic physical motion adhering to the plan.”

The researchers tested PlaMo in a series of simulations, using it to plan and execute the movements of the SMPL humanoid, a humanoid virtual agent with a neutral body structure (i.e., no facial features, hair, clothes, etc.). They ran their simulations on IsaacGym, a physics-based simulation environment for reinforcement learning developed at NVIDIA.

The results of these tests were highly promising, as the PlaMo approach was found to effectively plan and execute the movements of SMPL humanoids in complex simulated landscapes, following textual instructions. Notably, the approach’s planner component was found to account for the humanoid character’s movements on uneven terrains, as well as static and dynamic obstacles in the environment.

The motion controller was, in turn, found to reliably track the paths planned by the planner, executing complex and scene-aware movements for the humanoids, such as crawling under a low-hanging ceiling or rapidly moving to avoid an approaching obstacle.

Overall, the combination of the two modules was highly effective, producing realistic movements in response to changes in the environment.

“PlaMo focuses on planning and control,” Hallak, Dalal and their colleagues wrote. “It opens opportunities for combining it with modern language models and 3D scene understanding. With these expansions in mind, we see PlaMo as a stepping stone in a much greater system where non-playable characters (NPCs) are given roles to play, forming a rich simulated virtual world.”

© 2024 Science X Network