Unexpectedly, VR manufacturers rolled up for “a pair of hands” this spring.
In the first half of 2022 alone, four VR giants will improve the gesture interaction technology of their own VR devices.
On April 20, the gesture interaction version of Meta’s VR headset Oculus Quest 2 was iterated to 2.0, which solved the problem of hand positioning loss. On April 13, Pico, the VR brand owned by ByteDance, announced that its two enterprise-level VR headsets can use customized hand tracking accessories to enhance the gesture interaction experience. VR maker HTC also released a new wrist tracker in January to help predict the user’s hand position.
Oculus Quest 2 Gesture Interaction Version 1.0 and Version 2.0 Comparing Hand Fast Tracking
Whether it is bare-hand interaction technology or hand tracking accessories, gesture interaction technology for VR devices has been iterating continuously. At the same time, many VR companies are still trying to find new gesture interaction methods, such as VR gloves.
On May 6, Dutch VR glove manufacturer Manus announced the pre-sale of its new VR glove Quantum Metagloves with magnetic tracking technology. On May 4, the domestic motion capture technology giant Noi Teng released the Hi5 2.0 VR interactive gloves. As early as November 2021, Meta showed its own VR touch glove model and shared its latest achievements in VR gesture interaction.
From left to right are the VR gloves of Manus, Noitom and Meta
Not only that, but some potential VR players such as Apple, Google, and Microsoft have also been granted a number of patents related to smart rings , trying to make smart rings a key input device for next-generation gesture interactions.
Gesture interaction technology is the most basic and natural interaction method, and it is also one of the mainstream interaction methods in VR devices . Generally speaking, gesture interaction refers to the technology of recognizing human gestures through mathematical algorithms , including multiple processes such as image acquisition, gesture tracking, gesture recognition and analysis.
Among them, hand tracking and gesture recognition are two key steps in gesture interaction. Hand tracking refers to accurately locating the position of the user’s hands in the VR headset, and how to accurately identify the movements of each finger and various forms of the hand belongs to gesture recognition.
Gesture interaction technology also hides a huge market space. According to a report by Markets and Markets, an international market analysis company, the “gesture recognition and touchless perception” market will reach $ 34 billion in 2024 .
What are the new ways to play the gesture interaction technology of VR devices? Why are manufacturers upgrading the gesture interaction technology of their VR devices in the near future? What is the development of gesture interaction technology for VR devices?
We tried to find the answer after in-depth exchanges with Noitom CTO Dai Ruoli, Dream Bloom (iQIYI Intelligence) technical director Shi Ming, and HTC’s technical director.
Aside from the controller, VR games can be played with bare hands
Compared with other mobile devices, the technology of gesture interaction for VR devices is progressing slowly . For example, it has been nearly five years of development from 3dof to 6dof, and handle interaction to bare-hand interaction is still in its infancy. With the continuous advancement of technology, bare-hand interaction has also begun to popularize on many consumer-grade VR headsets.
Meta released version 1.0 of gesture interaction in 2019 and introduced bare-hand interaction into the Oculus Quest VR headset, allowing users to directly see their hands in the virtual world and perform some simple interaction operations. After a lapse of 2 years, Oculus Quest 2 has version 2.0 of gesture interaction.
This upgrade mainly solves two problems. One is to solve the problem that hand tracking is difficult when the user’s hands overlap. In version 1.0 of Gesture Interaction on the Quest 2, it was difficult for the system to recognize the user’s gestures when their hands were blocking or overlapping each other. Second, when the user waves his hands quickly, the system can easily lose track of the hands. Both of these situations can cause the hands to suddenly disappear from the virtual world.
Quest 2 gesture interaction version 1.0 and version 2.0 comparison
As for how to address both issues, Meta said in its announcement that the new version improves the performance reliability of hand tracking based on a “redesigned approach to computer vision and machine learning .” But since the version update, we have seen many users feedback that although the gesture interaction function has a better experience, the system also has obvious lags.
In response to this phenomenon, Dai Ruoli explained that the ability of VR headsets to process computing itself is relatively limited , while Meta acquires data based on computer vision, and uses part of the computing resources to predict the position of the user’s hand. VR headsets bring a lot of computational processing pressure , which is also the main reason for the “system lag” phenomenon.
HTC also launched a new wrist tracker VIVE Wrist Tracker in March this year, and upgraded its own hand tracking version at the end of last year. According to information on HTC’s official website, when the user wears the wrist tracker on the wrist, even if the hands are out of the camera’s field of view for a period of time, the system can predict the movement of the hand through the high-frequency IMU data and kinematic model of the wrist tracker. movement trajectory.
The relevant person in charge of HTC also shared with Zhixi that compared with the VIVE Focus 3 controller, the VIVE Wrist Tracker removes buttons, tactile input and other components to reduce the battery size and make the controller smaller and lighter. According to reports, HTC’s wrist tracker is 85% smaller and 50% lighter than the VIVE Focus 3 controller.
VIVE Wrist Tracker wrist tracker
We can see that Meta’s bare-hand interaction solution is still dominated by computer vision, while HTC’s bare-hand interaction solution needs to rely on peripherals such as wrist trackers. The former can help users experience the fun of bare-hand interaction quickly and effectively , while the latter improves the accuracy and reliability of gesture interaction to a certain extent through inertial sensors . Faced with different application scenarios, the bare-hand interaction solutions of VR devices from different manufacturers have their own emphasis.
But does bare-hand interaction necessarily bring people a better experience? Dai Ruoli said in an interview: “At a time when the technology has not yet reached maturity, embedding a new way of human-computer interaction such as bare-hand interaction into the operating system will give people too high expectations.”
“In terms of operation success rate and tactile feedback, handles are far superior to bare-hand interaction in terms of fun and operation accuracy.” He explained that in the field of human-computer interaction, 80% operation accuracy and 100% operation accuracy The interactive experience brought to the user is completely different.
Tong Xin, chief researcher of the Network Graphics Group at Microsoft Research Asia, confirmed this in an interview with the media in 2017. “When I hold the handle, the handle is not only an input device, but also an output device. It can give me an output feedback through vibration and force feedback. When I wave it in the air with my empty hand, I lose it. own output channel. ”
Limited computing consumption, high learning cost, and low operation accuracy are the main reasons that hinder the development of VR equipment bare-hand interaction technology. But in terms of interactive experience, bare-hand interaction will also provide VR content producers with more room to play . In many VR devices, bare-hand interaction is used as an auxiliary gesture interaction method.
The VR gloves that the big manufacturers are competing to make can be done for 100 yuan at station B?
In order to find a balance between the immersion brought by the bare-hand interaction and the high accuracy provided by the VR handle, VR manufacturers have come up with another way of gesture interaction – VR gloves.
At present, there are many forms of VR gloves, but the VR gloves that can be mass-produced and sold are mainly divided into two types : based on inertial sensors and based on bending sensors .
Different from the data processing method based on computer vision gesture interaction, the inertial sensor -based VR glove will collect a large amount of hand positioning and finger interaction data through the sensor. The VR gloves based on bending sensors have lower measurement accuracy, and the comfort of their wearable devices is also lower due to higher requirements for fit.
We had the opportunity to experience the Hi5 2.0 VR interactive glove first. A pair of Hi5 2.0 VR interactive gloves has a total of 12 inertial sensors , and each hand wears six wireless sensors, which are distributed on five fingers and on the back of the hand. At the same time, the set will also be equipped with a data transceiver, which can be plugged into a computer or VR headset to transmit data.
Hi5 2.0 VR Interactive Gloves
Turn on the sensor, install the sensor, fix the hand tracker, and select the calibration interface. The calibration method of the Hi5 2.0 VR glove is more complicated than we thought .
During the experience, we found that the Hi5 2.0 VR interactive glove can provide a more accurate gesture recognition process. For example, the position of the hand and the changes in gestures can basically display changes in real time. According to the official website, the frame rate of the device’s attitude calculation has been increased from 180fps to 500fps, and the developers have also improved a certain degree of anti-magnetic interference through algorithms.
Noitom R&D staff are calibrating the Hi5 2.0 VR interactive glove
In addition to this, researchers are also experimenting with different VR glove designs.
In November 2021, Meta announced its 7-year-old VR tactile glove prototype. This VR glove replaces the motor with a micro air valve, and adjusts the inflation device on the finger through a complex control system to form a realistic pressure to simulate various kind of touch.
Meta Touch Gloves
Dutch VR glove developer Manus’ Quantum Metatagloods VR glove is based on magnetic tracking technology.
Different from Noitom ’s solution that uses algorithms to reduce the magnetic interference caused by the natural environment and ensure tracking stability, the gesture interaction solution through magnetic tracking requires the cooperation of two devices. The VR glove will also contain a coil, and the two coils will generate magnetic lines of force in the form of high-frequency vibrations, and the two magnetic lines of force will cut each other to obtain the position change of the hand .
Although VR gloves can provide more accurate gesture motion capture, the price and cost of VR gloves determine that they can only be used by a small number of geek players and commercial markets.
It is reported that the prototype cost of Meta gloves is about 5,000 US dollars (about 33,400 yuan), while the pre-sale price of Manus’ Quantum Metagloods is 9,000 US dollars (about 60,000 yuan), and Noitom’s Hi5 2.0 VR interactive gloves are priced at 9,800 US dollars. Yuan Renminbi.
From left to right are the VR gloves of Manus, Noitom and Meta
Meta has described the project to develop a VR haptic glove as a “moon shot.” “When we started developing the VR haptic glove, we wanted to create a mass-produced, affordable consumer device that would allow people to experience the virtual world anywhere,” said Meta’s project leader: “But it’s almost impossible.”
During the investigation, we also found that many UP masters made VR gloves on station B. The cost of these gloves ranges from 100 yuan to 350 yuan, and some VR gloves even have a force feedback function. We have seen that these self-made VR gloves can be used in some simple VR games, and can make some simple gestures such as holding, pinching, and flipping, but the display delay is also relatively obvious, and some ghost scenes often appear.
UP owners of station B try their own VR gloves
Faced with the huge price gap of VR gloves, an industry insider of VR gloves analyzed that the cost of VR gloves for the B-end market is mainly in R&D investment, parts and human resources costs , and the price is determined according to market demand. This means that the price of making an ordinary VR glove may not be high, but making a pair of VR gloves that can be used in commercial scenarios and provide accurate precision requires a higher cost.
Although VR gloves can take into account the immersion of VR equipment and the accuracy of gesture interaction, how to reduce the price of VR gloves will be a difficult problem that must be solved in the development of VR gloves.
Want to have free “hands”?
Overcome difficult problems from algorithm, engineering and interaction design
From the perspective of bare-hand interaction and VR glove interaction, the development of gesture interaction technology for VR devices is constantly updated and iterative, and solutions have been gradually found for some of the difficulties faced before. But overall, there are still many difficulties in gesture interaction technology for VR devices.
What’s preventing us from throwing away VR controllers and adopting new gesture interactions? Shi Ming, the R&D personnel of the domestic VR head display adventure series and the technical director of Dream Bloom (iQiyi Intelligence), gave three key points in an interview with Zhishi.
The first is the algorithm level. At present, the gesture interaction technology used in VR devices basically adopts deep learning technology, so it needs to rely on a large amount of business scene annotation data during the research and development process. However, gesture-based 3D data acquisition costs are high, and it is difficult to ensure data quality and consistency.
Keypoint Recognition for Gesture Interaction
Second, gesture interaction technology is more complicated in engineering optimization. Gesture interaction technology needs to be based on an excellent model, and it needs to do small model inference and deduction. In the inference process, researchers must not only ensure the accuracy of the algorithm, but also ensure the real-time efficiency of the data.
The last is the design of interactive use. VR developers need to avoid visual blind spots when designing gesture interaction-related operations, and at the same time provide users with a better interactive experience.
Regarding the design of gesture interaction, Liu Xiaoyu, a former Facebook Reality Labs designer, also mentioned in an interview that gesture interaction technology is prone to occlusion during use, and the user’s learning cost must also be considered when designing gesture interaction actions.
In addition, Tong Xin, chief researcher of the Network Graphics Group of Microsoft Research Asia, also mentioned that for gesture recognition, it is difficult for machines to distinguish between input and non-input states . Different from the way the keys of the keyboard interact, it is difficult for the machine to distinguish whether you are shaking your hand, whether you are making a waving gesture, or whether you are tired of playing and want to shake your hand to rest.
In general, gesture interaction technology for VR devices as a whole mainly faces two challenges . On the one hand, which gesture interaction method can take into account immersion, accuracy and low cost, and can be used for all VR devices. On the other hand, how to create the most suitable gesture interaction function for this device according to the basic configuration of the VR device.
It is foreseeable that until the bare-hand interaction technology can achieve higher precision and the cost of VR gloves decreases, the handle will still be one of the mainstream solutions for gesture interaction of VR devices.
Conclusion: Gesture interaction technology may become the key to upgrading VR experience
At present, gesture interaction for VR devices is divided into three directions: the existing handle interaction method, the bare-hand interaction method that is gradually becoming the mainstream, and the interaction method of external wearable devices that are still in the niche stage. However, it is worth noting that these three interaction methods are not substitutes, but can achieve coexistence.
In the period of rapid development of mobile phones and computers, whether it is typing or the interaction mode of touch screen, it has brought new interactive experience to people. Gesture interaction will also bring new changes to VR device experience and VR content production.
The technological development of gesture interaction will always have endless directions, but the gesture interaction technology left by the times is still a minority. How will the gesture interaction technology for VR devices evolve? Maybe time will tell us the answer.
Posted by:CoinYuppie，Reprinted with attribution to:https://coinyuppie.com/vr-subversive-moment-throw-away-the-handle-and-enter-senseless-driving/
Coinyuppie is an open information publishing platform, all information provided is not related to the views and positions of coinyuppie, and does not constitute any investment and financial advice. Users are expected to carefully screen and prevent risks.