Tracking the same person even if the screen changes... UNIST develops 3D human body restoration AI

Artificial intelligence (AI) technology has been developed to consistently track the movements of multiple people and restore them in 3D even when the shooting angle and screen change in movies or sports broadcasts.

Ulsan National Institute of Science and Technology (UNIST) announced on the 15th that Professor Baek Seung-ryeol's team at the Graduate School of Artificial Intelligence developed 'Multi-THuMBS', a 3D human body restoration AI technology.

3D human body restoration is a technology that creates a three-dimensional human body model and movement by analyzing the body shape, posture, and position of the person in the video.

With existing AI technology, it was difficult to consistently restore people from videos that were shot from multiple angles.

When a person seen frontally appears in profile, the person is recognized as a different person, or the change in position on the screen is misinterpreted as actual movement.

The research team restored the background and camera positions just before and after the screen transition into one three-dimensional space, and then applied a method of arranging the human body model for each scene to fit this space.

Based on this, the distance between people, clothing, and posture were compared to connect the same people and smoothen their movements.

As a result of applying scene change editing to a test video of human movements recorded with multiple cameras, Multisomes' joint position error was the lowest among the compared technologies.

Compared to the next-best technology, errors were reduced by about 31%, errors in recognizing the same person as different people were reduced by about 62%, and camera position estimation errors were reduced by about half.

In actual movies and sitcom videos, the rate of matching joint positions before and after scene changes was 90.7%, which was higher than the 62.7% of the comparative technology.

Professor Baek said, “Even in edited videos such as movies, broadcasts, and sports broadcasts, the movements of multiple people can be restored in 3D. We expect it to be used for sports analysis, virtual and augmented reality content, and digital human production.”

Researchers from the University of Birmingham in the UK and CJ Corporation participated in this study, and it received support from the Ministry of Science and ICT, the National Research Foundation of Korea, the National IT Planning and Evaluation Institute, and CJ Corporation.

The research results were presented at the European Computer Vision Conference held in Malmo, Sweden last September.