Apple’s new patent for VR/AR: Using neural networks to simulate body movements, making VR wireless transmission faster

Apple’s new patent: Create your virtual “clone” by imitating your posture?

Apple's new patent for VR/AR: Using neural networks to simulate body movements, making VR wireless transmission faster

Apple has received two new patents, and VR headsets are about to meet?

Smart Things reported on July 14 that recently, two of Apple’s new VR/AR related patents have been authorized by the US Patent and Trademark Office (US Patent and Trademark Office).

The first patent is “Generating body pose information (Generating body pose information) system”. The system collects information from various parts of the body, and then uses this information to create a user’s avatar. The second patent is “Adaptive wireless transmission schemes system”. The system handles the communication between the VR head-mounted display device and the host.

It is reported that Apple is developing AR/VR headsets and AR smart glasses called “Apple Glass”. Apple is designing different solutions for the problems in the field of human-computer interaction, hoping that Apple’s head-mounted display products can stand out from other head-mounted display products.

01. The new system creates a “you” in the virtual world by collecting posture information

According to the patent report, Apple’s new patent “Generate Body Posture Information” system can collect user posture information through cameras, neural network systems, etc., to form a user’s virtual avatar.

1. Use neural network system to collect user posture information

Single neural network system can collect user posture information; a plurality of neural networks work together, you can simulate a joint user; finally, a summary of all information systems in all together they can create a user in the virtual world “cloned.”

In this process, different hardware devices require machine learning. Because the “generating body posture information” system not only needs to process the data collected by the camera, it also needs to “learn” information interaction between different neural network systems.

2. The neural network system covers all joints of the body

The neural network system collects information on which parts of the user’s body, and Apple gives the answer in the patent report.

Apple's new patent for VR/AR: Using neural networks to simulate body movements, making VR wireless transmission faster

▲Multiple neural network systems can monitor each joint of the user and jointly create a whole body model. (Source: Appleinsider)

The information collection scope of the “Generate Body Posture Information” system includes “neck, shoulder, elbow, wrist, pelvic, knee, ankle, and finger joints.” The system will not only define the scope of information collection for each part, but also determine where the connection between each part is.

Although the list of this patent gives all the basic elements needed to create a user’s body model, it does not seem to be very exhaustive. For example, the knuckles seem to only refer to the fingers, and the toes and foot movements do not seem to be paid attention to by the system.

3. Use data to create user avatars

Apple believes that creating a good immersive experience requires collecting as much information as possible about the user’s posture. Because VR or AR applications may need to present different content to the user based on the user’s static posture or dynamic behavior. For example, a guard in a VR game needs to react differently to the user’s different attack actions.

In fact, Apple hopes to use the “generate body posture information” system to control the user’s avatar in the virtual world. For example, virtual online social VR Chat may use this system to change the way the user’s avatar is presented.

02. Breaking through the communication barriers, the new patent uses the gaze function to allocate data resources

Apple’s second patent “Adaptive Wireless Transmission” system hopes to break the communication barrier between the headset and the host.

1. VR headset encounters communication problems

The current VR headsets on the market, such as Oculus Quest, although designers can increase the weight of the headset to reduce the weight of the connection between the headset and the host, the wiring of the connection itself is a problem.

In this regard, some people have proposed using wireless communication systems to reduce the weight of the headset and eliminate wires. But in fact, there are many problems in wireless communication systems.

For example, the wireless communication system needs to consume equipment power to transmit data, and the bandwidth of the wireless communication system is less than that of wired communication. Another example is the most common disconnection and interference from other equipment in wireless communication systems.

2. The new patent uses interlaced frame transmission

Faced with these problems, Apple’s new patent “Adaptive Wireless Transmission” system provides a new solution.

Although it uses a wireless communication system, it reduces the amount of data transmitted at one time. This scheme requires the system to transmit as little video data as possible. In addition, this patent chooses to halve the bandwidth of the network and achieve interleaved frame transmissions through the interleaved use of left and right eye data.

Apple patent that new data does not have to transmit each picture a . It divides the transmission frame of the picture into different categories, one is the picture frame that needs to be updated urgently, and the other is the picture frame that can be used again, is not updated, and is at an earlier time.

Apple's new patent for VR/AR: Using neural networks to simulate body movements, making VR wireless transmission faster

▲Alternating video frames for each eye can halve the amount of data transmitted at one time. (Source: Appleinsider)

3. Gaze tracking function supports data throttling

In addition, the patent also uses the gaze tracking function (mention of gaze tracking). This function can give priority to updating the data of the screen that the user is viewing through the user’s gaze. Not only that, also by injection -view display technology (foveated display techniques) to reduce the amount of detail screen the user is not concerned about saving data transfer again.

But implementing this scheme may require extremely high frame rates. According to an Apple patent report, because the headset is in the eye area, “at least 100 frames per second”.

03. Technology will continue to iterate, but not all of it will be applied to Apple products

Prior to this, Apple had many patent applications in the field of VR and AR, and there were some technical overlaps with this two new patents.

For example, in terms of foveated displays, the foveated area is a key area of ​​the eye, which enables people to see things in a very clear and precise way. In 2020, Apple designed a system similar to this one. The system can use eye tracking to determine where to place data resources on the screen, and it can be ensured by using gaze detection. Whether the screen has been updated.

Another example is the gesture recognition function. In 2018, an Apple patent involved how to use the camera under the headset to monitor the user’s gesture changes so that the system can manage 3D documents through gesture changes. In 2015, an Apple patent can recognize people’s gestures from a distance through machine vision.

Apple submits a large number of patent applications every week, but the existence of new patents only means that Apple’s R&D team is interested in this field, and does not guarantee that these patents will appear in Apple’s products.

04. Conclusion: Will Apple bring VR/AR devices on fire?

Two new patents from Apple in the field of VR/AR reveal to us the fact that Apple already has a relatively mature solution to the difficult problems in the field of human-computer interaction.

According to a report by Tianfeng Securities analyst Ming-Chi Kuo, Apple may launch VR/AR headsets in the middle of 2022. He believes that the success of the Apple headset depends on whether the VR/AR application can provide a very smooth display and operation process.

According to IDC, by 2022, the market value of VR/AR is approximately US$56 billion. When Apple enters the market with head-mounted display devices, whether it can accelerate the landing of VR/AR devices through new technologies, we are watching the changes.

Source: AppleInsider

Posted by:CoinYuppie,Reprinted with attribution to:https://coinyuppie.com/apples-new-patent-for-vr-ar-using-neural-networks-to-simulate-body-movements-making-vr-wireless-transmission-faster/
Coinyuppie is an open information publishing platform, all information provided is not related to the views and positions of coinyuppie, and does not constitute any investment and financial advice. Users are expected to carefully screen and prevent risks.

Leave a Reply