An article to understand the motion capture technology: one of the “important puzzles” of the Metaverse

Motion capture is a key step in “digitizing” people

If the Metaverse is the ultimate form of “digitization”, then motion capture technology is a key step in realizing the “digitization” of people.

An article to understand the motion capture technology: one of the "important puzzles" of the Metaverse

In film and television production, motion capture is one of the most commonly used techniques. Whether it is “Avatar” or Gollum in “The Lord of the Rings”, it is first used to capture the physical performance of the actor, and then the captured action rendering processing is presented before presenting a shocking visual effect.

The game industry is also the core application scenario of motion capture technology. The game animation contains a lot of complex pose movements, by collecting the action data of real actors, binding to the bones of the game characters, you can maximize the restoration of the real posture, expression, weight and speed of the human body, so that players can experience a more realistic game world.

With the full popularization of the concept of “Metaverse”, the long-term value of motion capture to the Metaverse has gradually emerged, which is at the same level as the engine, transmission, calculation and display technologies, and is an important piece of the “huge puzzle” in the underlying construction of the Metaverse.

The development of motion capture technology

Motion-capture techniques first appeared in 1915, when the animation master Max Fleischer built a projector based on the principle of displaying the contents of the film on a transparent table. With this projector, animators can easily follow the action of the characters in the picture to draw the character movements.

An article to understand the motion capture technology: one of the "important puzzles" of the Metaverse

Figure | Painting projector

In 1983, Tom Calvert of Simon Fraser University in Canada made a major breakthrough in physical mechanical capture clothing, a technology that gave people a glimpse of the earliest mechanical capture. At the same time, MIT also launched a set of LED-based “graphical marionetter” systems, which were the prototypes of early optical motion capture systems.

An article to understand the motion capture technology: one of the "important puzzles" of the Metaverse

Figure | Tom Calvert’s research

This biomechanical research paved the way for future filmmaking, and in the following time, when motion capture and computer graphics technology met, the ease of access to motion data led to the rapid development of motion capture technology, and has been fully and massively applied to the game and film industry.

At the end of the 90s, the filming of the movie “The Lord of the Rings” brought the shooting steps of the motion capture to the shooting scene for the first time, and the pioneer of the motion capture actor Andy Serkis can interact with other actors as the role of “Gollum” in the venue, which is more conducive to the shaping of the role, because only when the actor obtains the emotional and linguistic feedback of other actors during the performance, his emotions can be released more smoothly, and the role can be more flesh and blood and vivid.

An article to understand the motion capture technology: one of the "important puzzles" of the Metaverse

Figure | Motion capture stills from the movie The Lord of the Rings

The film “Avatar”, released in 2009, can be said to be a pioneer in the successful combination of motion capture and expression capture technology. Director James Cameron and his team used a head-mounted facial capture camera and built the largest shooting and motion capture studio ever built.

An article to understand the motion capture technology: one of the "important puzzles" of the Metaverse

Figure | Motion capture stills from the movie Avatar

Special effects film and television production and games have never been separated, and soon someone brought the concept of motion capture to the game circle. The most pioneering in this field was Sega, who was in the console field at that time and Nintendo and Sony.

Its 1994 arcade game VR Warrior used motion-capture simulation of character action. This new concept became a clear stream in the rough arcade and home console gaming market at the time, scaring a large number of players with realistic and smooth movements. The following year, Namco also launched “Soul of the Sword”, which was also successful as the vanguard of its own motion capture technology.

Nowadays, motion capture is almost standard in large game studios, and with the use of motion capture technology, live-action and animated characters are synchronized, and the game characters will appear more realistic and vivid. That’s why we can see cinematic level action performances in the game.

Common motion capture techniques

With the maturity of technology, the application of motion capture technology is now more and more extensive, from animation production, human-computer interaction, to robot remote control, sports training, etc., and even the current virtual human live broadcast, is also the use of motion capture technology.

In the face of different use scenarios, motion capture technology has also appeared a variety of technical routes, common optical motion capture technology, inertial motion capture technology and visual motion capture technology.

When the optical motion capture technology is operated, it will be directly marked on the human body, and the marking point will be directly reflected to the pre-set camera, and then the spatial movement information of the marking point will be budgeted by the imaging information of different locations reflected, and finally the information will be simply located and output.

An article to understand the motion capture technology: one of the "important puzzles" of the Metaverse

Figure | Optical motion capture: Marking the light spot on the body

Inertial motion capture technology will wear a gyroscope directly on the person’s body, and when the person is moving, the gyroscope will also rotate with it. At this time, the human movement is directly calculated by perceiving the rotation information of the gyroscope, and then the motion capture is realized.

An article to understand the motion capture technology: one of the "important puzzles" of the Metaverse

Figure | Inertia motion capture requires wearing a variety of devices

Visual motion capture technology does not need to mark and wear equipment when operating, as long as the action is recorded through ordinary cameras within the range of human activity, the key information of the human body is recognized, and then special AI algorithms are used to achieve motion capture.

An article to understand the motion capture technology: one of the "important puzzles" of the Metaverse

Figure | AI engine-driven motion capture technology

Optical motion capture technology and inertial motion capture technology has a certain threshold for use, which is more common in the field of film and television and games, although the effect presented is very accurate, but there are two problems: First, the cost is high. Cheaper ones also need at least tens of thousands, and expensive ones need hundreds of thousands to millions, and only large film and game studios can afford this cost. Second, it is inconvenient to use. On the production site, motion capture actors often wear a lot of equipment, and wearing devices and motion capture requires the cooperation of multiple teams.

The visual motion capture technology, which is more convenient for popularization in the general consumer market, has been chased by major manufacturers such as Apple and Meta in recent years.

Meta uses a headset to do a full-body motion capture

As early as 2019, Meta announced its virtual human avatar system, which is characterized by 3D motion capture technology through VR equipment to restore real people’s images, which can render highly fidelity skin tones, textures, hair, micro-expressions and other details. Meta wants people to meet in a virtual environment in the future as real as they are in reality.

An article to understand the motion capture technology: one of the "important puzzles" of the Metaverse

Figure | Quest, a VR device owned by Meta, recognizes facial expressions

According to a paper published this month, Meta proposed a solution for full-body motion capture through Quest alone. In other words,VR headsets can only capture facial expressions for motion capture, but now they can achieve full-body motion capture.

This is largely driven by the predictive power of artificial intelligence.

For upper body tracking, the experience gained during AI training requires only a small amount of input from the real world to accurately convert both hands to the virtual world. For example, Quest’s camera can see your arms, elbows, palms, so it’s a good idea to estimate the full posture of your upper body based on musculoskeletal structure.

An article to understand the motion capture technology: one of the "important puzzles" of the Metaverse

Figure | The Quest headset enables full-body motion capture

Now for the lower body, Meta is also exploring the use of this principle. Train AI using collected tracking data to realistically animate a full-body virtual person using only sensor data from a VR headset and two controllers.

The Meta team trained QuestSim (AI engine) using human-generated sensor data. To do this, the researchers simulated the motion of the headset and controller based on 8 hours of motion capture clips from 172 people each. This way, they don’t have to capture data on headsets and controllers with body movements from scratch.

Motion capture clips include 130 minutes of walking, 110 minutes of jogging, 80 minutes of gestures, dialogue, 90 minutes of whiteboard discussion, and 70 minutes of balance.

An article to understand the motion capture technology: one of the "important puzzles" of the Metaverse

Figure | AI engine in self-learning

After training, QuestSim can identify the actions a person is performing based on real headset and controller data. Using artificial intelligence predictions, QuestSim can even simulate body part movements without real-time sensor data.

The researchers further found that even without the controller of the handle, only 60 poses of the headset (including position and orientation data) were sufficient to reconstruct various movement gestures, and the resulting effect was also free of physical artifacts (imaging that did not exist but appeared in the image).

For the future of motion capture technology, CITIC Securities believes that motion capture technology is expected to be further developed and applied in the direction of biomechanics, engineering applications, games, film and television, and VR. In the process of the development of the Metaverse, capturing user actions and generating corresponding performances in the virtual world in time is an important part of the user’s high-quality experience, and there will be a wide range of basic application space for motion capture in the future.

Posted by:CoinYuppie,Reprinted with attribution to:https://coinyuppie.com/an-article-to-understand-the-motion-capture-technology-one-of-the-important-puzzles-of-the-metaverse/
Coinyuppie is an open information publishing platform, all information provided is not related to the views and positions of coinyuppie, and does not constitute any investment and financial advice. Users are expected to carefully screen and prevent risks.

Like (0)
Donate Buy me a coffee Buy me a coffee
Previous 2022-09-30 10:58
Next 2022-09-30 22:08

Related articles