Meta throws out 10 hard-core technologies to reveal Zuckerberg’s Metaverse ambitions

The ultimate VR, looks like this? !

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

Speaking of cutting-edge research progress in the field of VR (Virtual Reality), Meta, the world’s most popular VR equipment supplier, definitely leads the market. 

Meta CEO Mark Zuckerberg recently revealed Meta’s latest layout in the Metaverse in an interview.Meta will launch a new VR headset, the Meta Quest Pro, in October. 

It is reported that the biggest highlight of the headset is that it has face and eye tracking capabilities, which can allow virtual digital avatars to make eye contact. Hardware analyst Brad Lynch believes that the Meta Quest Pro will use an improved XR2 processor, which will provide a 30% year-on-year performance increase over the existing XR2 processor. Analyst Ming-Chi Kuo predicts that the Quest Pro will use a single-eye 2160×2160 resolution panel. 

In the interview, Zuckerberg also talked about social Metaverse. He believes that from text, images, to video, the carrier of social media is evolving towards a larger information capacity, and the next one can carry more than video. The carrier of the amount of information is in the field of VR.

Earlier, Zuckerberg personally recommended his own social platform Horizon Worlds and posted a VR selfie. Although the photo was ridiculed by the entire Internet group, Zuckerberg still brought a wave of Internet popularity to Horizon Worlds. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲ Zuckerberg’s VR selfie

Meta has been making a lot of moves in the Metaverse of late. In June of this year, Reality Labs, Meta’s VR division, launched four VR headset prototypes, showing four key technological evolution directions of future VR devices: high resolution, high dynamic range (HDR), thin and light form factor, and reduced dizziness. Sensitive zoom technology optimization. At the SIGGRAPH conference held in August, Douglas Lanman, head of display system development at Meta, gave a speech and introduced ten challenges to breaking the bottleneck of VR technology. 

At the SIGGRAPH 2022 conference, foreign media Auganix reporter Sam Sprigg specially experienced the Starburst, a VR prototype that features HDR. The Oculus Quest 2 is the pinnacle of Meta’s all-in-one VR headset so far, with shipments exceeding 10 million units last year, making it the best-selling VR product in history, and Starburst’s peak brightness is 200 times that of the Quest 2. 

So, what exactly is the Metaverse layout of Meta? How did the Meta prototype feel like? Where has Meta’s tens of billions of R&D been invested? Let’s take a look at Meta’s latest developments in the VR field with these questions in mind. 

01. In order to explore four research areas, Meta released four prototypes

In June, Meta Reality Labs unveiled four prototype VR headsets in different technology fields. 

Butterscotch explores resolution technology, Starburst explores HDR technology, Holocake 2 explores thin and light form factor technology for VR headsets, and the Half Dome series explores zoom technology. The four prototypes represent the latest advancements Meta has made in the development of four VR headsets. 

Previously, Meta CEO Mark Zuckerberg proposed the concept of the Visual Turing test. To put it simply, the evaluation standard of the visual Turing test is whether the virtual image can be confused with the real one, so that the human eye cannot distinguish the virtual image from the real image.Zuckerberg said the visual Turing test has two focuses, photorealism and presence. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲Comparison of different VR headset resolutions

One of the ways to make an image more realistic is to increase the resolution of the image as much as possible. To this end, Meta developed a prototype Butterscotch, which features “retina-level” resolution. 

Butterscotch’s PPD reaches 55, which is very close to the “retina resolution” level of 60 pixels per 1 degree angle, which is 2.5 times the resolution of the Quest 2, said Yang Zhao, an optical scientist on the DSR team in the research division of Meta Reality Labs. Butterscotch’s custom lens model In the group, there are two high-precision glass lenses, which have a higher surface quality but are also heavier. In addition, there is a special lens that diffracts light. The combination of the glass lens and the diffractive lens can present a clear image. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲Starburst prototype

The Starburst prototype features high HDR (high dynamic range imaging) capabilities, with a backlight that can reach a maximum brightness of 20,000 nits, which is 200 times that of the existing VR headset Quest 2. The Meta Reality Labs team developed Starburst to restore full indoor lighting in a virtual environment and to explore the ideal brightness for the user to achieve the best immersion experience. 

Starburst replaces the original LCD screen backlight with COB LEDs (LEDs commonly used in car lights), which greatly improves the LCD backlight brightness, but also generates more heat.Therefore, the Starburst also comes with a fan and a 3D printed cooling chassis. However, the overall device is relatively bulky and consumes a lot of power. Of the four technologies, this one is the furthest from the market. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲Holocake 2 prototype

The Holocake 2 prototype is characterized by being thin and light, which can improve the wearing comfort of the VR headset. Holocake 2 is a PC VR device. Although its shape is thin and light, it requires additional space for computing and battery components and adopts a split design. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲Half Dome development history

Half Dome is a prototype VR headset based on the Oculus Rift, featuring a zoom lens. Starting from the third generation, the lens of Half Dome has been upgraded from a mechanical system to an electronic zoom system, reducing mechanical noise and vibration. Half Dome 3 has 32 switchable focus planes. According to experiencers, when playing games such as “Lucky’s Tale” through Half Dome’s dynamic zoom, the focus of the image in the field of view can change according to the distance the controller moves. 

02. The peak brightness of Starburst is 20000nit, which is 200 times that of Quest 2

At SIGGRAPH 2022, the top computer graphics conference held this month, Sam Sprigg, a reporter from foreign media Auganix, specially experienced the Starburst, a VR prototype that features HDR.The Oculus Quest 2 is the pinnacle of Meta’s all-in-one VR headset so far, with shipments exceeding 10 million units last year, making it the best-selling VR product in history, and Starburst’s peak brightness is 200 times that of the Quest 2. 

At the exhibition site, two Starbursts were hung on a frame for participants to experience. The Starburst weighs about 2.5kg, which is a bit too big for a headset. “It’s so heavy that you can’t put it on your head, you have to hold it up to your face,” says Sam Sprigg. And, the Starburst needs a lot of power to power its high-brightness display, so it’s affixed to the frame. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲ Two Starbursts are hung on a shelf

The brightness of the Starburst is measured in “nits,” and Sam Sprigg claims that the brightness of the indoor environment exceeds 10,000 nits, while the peak brightness of the Starburst is 20,000 nits.Meta has said that the peak brightness of the Quest 2 is about 100nit, and the peak brightness of the Starburst is 200 times that of the Quest 2. 

And the peak value of 20000nit can also explain the “clunky” of Starburst to a certain extent. High brightness also generates high heat, and to dissipate heat, the Starburst is equipped with a large aluminum heat sink. Aluminum heat sinks make up at least 50 percent of a device’s weight, says Nathan Matsuda, a research scientist in Meta’s Display Systems Research (DSR) group.Additionally, there’s a large fan on top of the unit that absorbs heat from the heatsink. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲ Starburst consists of 36 parts

However, the Starburst has a lower resolution at around 2560×1620 pixels, which is slightly lower than the Quest 2. However, this is also acceptable. Because Starburst mainly shows HDR solutions, as for high resolution, that is the direction that Butterscotch should explore. 

Starburst uses both color and monochrome LCDs (liquid crystal displays). That’s because a single LCD has poor contrast in low light, while two color LCDs absorb most of the light, degrading the display, Matsuda said. Therefore, the best display effect can be obtained by using color and monochrome LCDs in series. 

In order to focus on exploring HDR technology, Starburst made compromises in the refresh rate and field of view of the picture. However, Sam Sprigg said that this compromise did not affect the Starburst’s performance in terms of HDR and brightness. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲ Sam Sprigg “wears” Starburst

The first part of the Starburst demo is to set up a light source in virtual space and place some spheres to show the reflected light. Sam Sprigg said that at 10,000nit display range, this is the same experience as the Quest 2, there is nothing special about it. 

However, when the brightness rises to 10000nit, he can feel more obvious brightness. When showing 10000nit of light, the demo switched to an ocean sunset view. The sun pierced through the clouds and the light hit the waves of the ocean, making the water sparkle. Then, the light rose to a peak of 20,000nit, and there was a virtual sun in the picture. Sam Sprigg said he felt as if he saw the real sun because the light was so bright. Finally, as the sun went down, the waves gave off a faint reflection.This demo is impressive. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲Place the sphere test light source

According to Sam Sprigg, staring directly at a 20,000nit light source in Starburst may feel like staring at a 60-watt light bulb. However, the light source in VR is only inches from his eyes, which is close. He believes that this brightness display can bring users a more realistic and immersive experience. 

However, Sam Sprigg also pointed out that power, heat, and form factor are important obstacles that limit the development of HDR technology for VR headsets. He believes that it may take a long time for us to wait for a mature product that can output 20,000 nits. 

03. There are many difficulties, and the VR headset faces ten technical challenges

At present, although Meta has made some technical progress on VR headsets, it still encounters many technical bottlenecks that cannot be solved at present. In his speech at SIGGRAPH, Douglas Lanman, head of Meta display system development, provided us with some reference directions – there are ten challenges that need to be solved urgently to take the VR experience to the next level! 

Whoever can take the lead in breaking these challenges is expected to gain the upper hand in the increasingly fierce Metaverse Hardware Entry Competition. 

1. Resolution: Meta develops a prototype Butterscotch to improve resolution

To increase user immersion and make virtual worlds look more realistic and clear, VR headsets must have higher resolutions. The resolution of current VR headsets is still very far from the level of human vision. 

Meta said their initial goal is to achieve 8K resolution per eye and 60PPD (pixel density). But at present, Quest 2 has only achieved 2K resolution and 20PPD per eye, which is still a long way from this goal. 

In order to increase the resolution of VR headsets, Meta developed a Butterscotch prototype focused on exploring resolutions. In this regard, the development and production of high-resolution displays is not difficult, the difficulty lies in how to equip high-resolution displays with the corresponding computing power. Lanman pointed out that two technologies, Foveated Rendering and Cloud streaming, may help to improve computing power, but the development of these two technologies is also very difficult. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲The above is the VR effect of ordinary blur processing, and the bottom is the blur effect of foveated rendering technology

2. Field of view: The field of view of the VR headset is twice as small as that of the human eye

The average human horizontal field of view is 200 degrees, while most current commercial VR headsets have a horizontal field of view of 100 degrees. This means that the field of view of a VR headset is narrower than what we see in a real environment. In addition, in terms of vertical field of view, VR headsets also have room for improvement. 

The wider the field of view, the more pixels the VR headset can render. This requires higher power and more waste heat. In addition, when the line of sight is widened, the image at the edge of the line of sight is easily distorted. Therefore, a wider line of sight requires better lens technology support.And, the researchers also want to make sure that the lenses and display are built in a way that doesn’t increase the VR headset’s form factor. 

3. Ergonomics: VR headsets are not comfortable enough to wear

Most current VR headsets tend to be slightly “clunky,” such as the Quest 2, which weighs over 500 grams and has a device thickness (protruding from the face) of 8cm. Such a device will obviously not bring a good wearing experience to the user. The ideal VR device should be thinner and lighter, allowing users to wear it comfortably for long periods of time. 

Pancake lenses and holographic lenses may help reduce the size of the headset, and Meta’s Holocake 2 prototype is currently exploring this direction, but it has not yet developed to the level of mass production. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲ “Sunglasses” is a prototype of the Holocake 1, and the transparent one is the Oculus Rift in 2016

4. Vision correction display: No low-cost vision correction display has yet been found

Better VR headsets should have vision correction capabilities so users don’t have to wear regular glasses under the VR headset. This problem can be solved with special lens attachments, but this is not an ideal solution. The best solution is for the headset’s display to have its own vision correction function, so that the headset designer no longer needs to consider issues such as fitting glasses. 

But the challenge is how to find a solution that is cheap to manufacture and doesn’t add extra weight to the headset? 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲ Lens accessories with vision correction functions already exist, but they are not the ideal solution

5. Zoom: Meta prototype Half Dome can achieve human eye zoom

In the natural environment, in order to see pictures at different distances, the human eye can naturally zoom. But in VR headsets, because the distance between the human eye and the screen does not change, the human eye cannot zoom or see the difference between near and far objects. There is a vergence-accommodation-conflict problem. So sometimes users experience eye fatigue, even headaches and vomiting. 

To solve this problem, researchers at Meta have developed Half Dome, a series of prototypes that support “progressive vision.” Half Dome can simulate different focal planes, showing different degrees of blur, thus helping the human eye to adjust the focus in the virtual world. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲The pupil will change with the distance of the object

6. Eye tracking: it is difficult to adapt to everyone’s eye conditions

Eye tracking is a key technology in virtual reality. It is the basis for many other important VR technologies, such as zoom, distortion correction, etc. Through this technology, users can achieve eye contact with other users in the virtual world, and have a more realistic social experience. 

The problem is that everyone has a different pupil shape, and everyone’s eyelids and eyelashes grow differently, so current eye-tracking technology isn’t well suited for all users. Therefore, the Lanman team will extensively collect more user data to upgrade the eye tracking technology and strive to make it suitable for more people. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲The shape of the pupil varies from person to person, which is a challenge for eye tracking technology

7. Distortion correction: Meta developed a distortion simulator to speed up algorithm iteration

The movement of the pupil can distort the image, thereby reducing the user’s sense of immersion, especially when used with zoom technology. To develop the correction algorithm, researchers must test the adjustment on a physical headset, but producing headsets can take weeks or even months, significantly delaying algorithm iterations. 

To solve this problem, the Meta research team developed a distortion simulator. With this simulator, researchers can test correction algorithms without producing test headsets and special lenses. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲The red light is uncorrected, the blue light is the corrected result

8. High Dynamic Range (HDR): The peak brightness of the Starburst prototype is 20000nit

Physical objects and environments are much brighter than VR headsets, and to address this, Meta created the Starburst prototype. The peak brightness of Starburst is 20,000nit, which is 200 times that of the existing Quest 2. 

Starburst can more realistically simulate enclosed spaces and lighting conditions at night, making virtual environments look more realistic. However, currently the Starburst is heavy and consumes a lot of energy. 

Meta believes that HDR contributes more to visual realism than resolution and zoom, but the technology is the most distant from practical application. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲ Close-up of Starburst prototype

9. Visual realism: Passthrough technology is not yet mature

Better VR headsets should also allow VR users to stay engaged with the real world. VR headsets can record real-world environment information and display it as video images in virtual reality. This technique, called “passthrough,” has been implemented in commercial VR headsets, but the quality is poor. For example, the Quest 2 offers a black-and-white perspective mode, while Meta’s upcoming high-end VR headset “Quest Pro” aims to pursue higher resolution and color perspective capabilities. 

However, VR headsets have yet to achieve a perfect reconstruction of the physical environment. The perspective of the picture captured by the perspective technology is spatially deviated from the eye, and users may feel uncomfortable when using it for a long time. So, Meta is working on AI-assisted line-of-sight synthesis, which generates perspective-correct viewpoints in real-time with high visual fidelity. 

In addition, Meta is also working on a “Reverse Passthrough” feature that would allow real-world users to see the VR user’s glasses and face to make eye contact. But currently the technology is visually incongruous, unnatural, and far from ready for the market. 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲ On the left is an Oculus Rift, which blocks the view of VR users, and on the right is an early reverse “see-through” prototype

10. Facial reconstruction: Meta conducts research on “virtual avatar” project

Meta hopes that in the future people will meet in a virtual environment as real as in reality. To that end, Meta is working on a research project called Codec Avatars, which is dedicated to creating 3D digital avatars for people in virtual environments. 

Currently, VR headsets can read the facial expressions of VR users in real time and transfer them into virtual reality. It is reported that the upcoming Quest Pro may be the first device from Meta to offer face tracking . 

Meta throws out 10 hard-core technologies to reveal Zuckerberg's Metaverse ambitions

▲ Quest Pro can recognize facial expressions and transmit them to VR devices

04. Conclusion: Meta’s VR technology has made breakthroughs but still needs to continue to overcome technical difficulties

Lanman presented the top ten challenges currently facing VR headsets at SIGGRAPH 2022, and we can see the progress and unsolved problems of the current Meta. Meta has released four prototypes that aim to solve the four problems of VR headsets in terms of resolution, zoom, thin and light form factor, and HDR. These four prototypes have their own strengths and have achieved preliminary technical progress in their respective fields. In addition, Meta has carried out a virtual avatar project to assist the facial reconstruction function of VR headsets, and developed a distortion simulator to accelerate algorithm iteration. 

These prototypes show the highest level of technology that Meta can achieve in VR today, but these technologies are far from enough to create a full-fledged VR headset. At present, there are still some technical problems that Meta cannot overcome temporarily. For example, the field of view of the VR headset is narrower than that of the human eye, the wearing comfort of the VR headset is not enough, the VR headset is not equipped with vision correction function, the eye tracking technology of the VR headset is not suitable for everyone, and the VR headset is not suitable for everyone. The “see-through” technology is not yet mature and so on. 

Counting the current exploration of Meta on VR headsets, we can find that Meta has come a long way on the way to explore VR headsets. However, Meta is far from the end of this road. At present, various technologies related to VR headsets are under active development, but these technologies are far from mature, and there is still a long way to go before they are commercially available. From this perspective, the explosion of VR is still far away. 

Source: Auganix, Mixed 

Posted by:CoinYuppie,Reprinted with attribution to:https://coinyuppie.com/meta-throws-out-10-hard-core-technologies-to-reveal-zuckerbergs-metaverse-ambitions/
Coinyuppie is an open information publishing platform, all information provided is not related to the views and positions of coinyuppie, and does not constitute any investment and financial advice. Users are expected to carefully screen and prevent risks.

Like (0)
Donate Buy me a coffee Buy me a coffee
Previous 2022-08-29 11:13
Next 2022-08-29 11:15

Related articles