From “His Royal Highness” Hatsune Miku, who was popular in the subculture circle ten years ago, to the current Douyin celebrity “Liu Yexi”, Netease Yanxuan’s spokesperson “Lin Yaoyao”, and scenes that are spread across major social platforms, shopping malls, etc. Chinese virtual anchors, virtual idols, virtual employees… It is undeniable that virtual people have broken the boundaries of once niche groups and are infiltrating people’s work and life with lightning speed.
How hot is this market? According to data from market analysis agency iiMedia Research, in 2021, the market size and core market size of my country’s virtual human-driven industry will be 107.49 billion yuan and 6.22 billion yuan respectively, and it is expected to be 640.27 billion yuan and 48.06 billion yuan respectively by 2025. strong.
As the 5G infrastructure has been fully rolled out, the acceleration of 5G applications has catalyzed the rise of new digital economic formats such as live e-commerce, digital entertainment, and new media, which has also brought about a large talent gap. Taking the live broadcast industry as an example, the shortage of live broadcast talents in my country will reach 8 million in 2021, and this number is expected to soar to 19.415 million in 2025.
Based on this pain point in the industry, many companies and MCNs have begun to try to fill the market gap with virtual human technologies such as virtual anchors and digital employees, and at the same time meet the diverse needs of users. Virtual humans are gradually playing an increasingly important role in people’s work and life. Role. Under this huge market demand, more and more players are pouring into the virtual digital human market, and the whole is still in a period of melee and growth.
Among them, NetEase, as one of the core players of the virtual digital human race, established the first domestic game artificial intelligence research institution – NetEase Fuxi in 2017. The original intention was to create a new generation of players through the use of artificial intelligence (AI) technology. Game experience, promote the development of AI technology, research directions include reinforcement learning, virtual human, user portrait, robot, cloud computing platform, etc.
In order to further understand the technical barriers and business logic behind NetEase Fuxi virtual human, 36氪 recently had an in-depth exchange with Li Feng, technical director of NetEase Fuxi virtual human, and Lao Yang, product director of NetEase Fuxi virtual human, and also gained insights in the process. At the moment when the new format of the digital economy and the concept of the Metaverse are breaking out, how NetEase Fuxi has deeply cultivated the application highland of virtual human.
NetEase Fuxi Studio
01. Two business directions of the Fuxi virtual human team
In fact, virtual digital humans are not a new technology. As early as 1992, American writer Neal Stephenson (Neal Stephenson) created two concepts of Metaverse and Avatar in his science fiction novel Snow Crash. In a flash, 30 years have passed, and now the concept of the Metaverse is on fire, and virtual digital humans are also considered to be the entrance to the Metaverse in the future.
The birth of NetEase Fuxi came earlier than the “Metaverse fever”. Since its birth in 2017, it has accumulated rich AI technical capabilities, published more than 100 top conference papers, and has more than 200 invention patents.
NetEase Fuxi’s virtual human business can be roughly divided into two parts. The first is to focus on the Leihuo business group to provide artificial intelligence virtual human technology services for games. As more and more game developers attach importance to visual presentation, it has become a broad consensus to produce high-quality plot animations for games. However, traditional animation production has strict requirements, high degree of specialization, high production cost, and long production cycle, which has hindered the process of high-quality production to a certain extent.
In this regard, NetEase Fuxi has launched an AI plot animation production tool. The core ability is to automatically synthesize highly expressive game character expressions, actions, and voices by entering a line, and supports Chinese, English, Japanese, Korean and other languages. Up to now, the AI plot animation production tool has provided services for games such as “Reverse Water Cold”, “Heavenly Mandate”, and “Eternal Calamity”, among which more than 10 hours of plot animations have been produced for the “Heavenly Mandate” mobile game.
“Heaven’s Mandate” mobile game plot animation
Secondly, with the continuous maturity and accumulation of team technology, under the explosive market demand of the virtual human industry, the Fuxi virtual human team began to commercialize To B, positioned the virtual human technical service team, and provided standardized SaaS platform services and SDKs to the outside world. products, and actively expand ecological partners.
Specifically, Fuxi virtual human external business can provide two sets of AI virtual human technology service combination, virtual avatar and virtual intelligent body. Among them, the virtual avatar includes intelligent face pinching + 3D animation migration, which mainly creates virtual world avatars for users and realizes real-time interactive gameplay; virtual agents include NLP dialogue + 3D animation synthesis + speech synthesis, which can mainly create intelligent NPC assistant customer service etc., to meet the service requirements of business scenarios.
For example, the 3D animation migration technology in the virtual avatar can complete the calculation of expression, action, and gesture parameters within 40ms according to the image input by the user, and drive the virtual character created by the user through intelligent face pinching in real time, so as to provide the user with real-time performance in the virtual world. Changyou provides basic interactive services.
In addition, NetEase Fuxi’s art research and development capabilities should not be underestimated. Based on NetEase Leihuo’s professional art team, Fuxi’s virtual human business can provide full-process, high-quality art performance to improve optimization capabilities, with styles covering Q version, beauty, realistic digital human, etc., and also has professional-level motion capture equipment, scanning Arrays that support model and action style customization.
NetEase Thunder Fire Motion Capture Center, Scanning Center
This series of core capabilities also provides more room for imagination for the future commercialization of Fuxi’s virtual human business. Among them, NetEase Fuxi’s image design and AI-driven services can provide services for customers who need IP images to improve their expressiveness, or who want to drive avatars through actions and videos, and work with partner companies to create industry benchmarks for virtual human + scenes Case, providing exclusive virtual IP customized full-link solutions for cooperative enterprises.
Today, Fuxi virtual human technology has been implemented in the fields of games, entertainment, cultural tourism, e-commerce, fast-moving consumer goods, education and other fields, providing AI virtual human for McDonald’s, Pepsi, Super Dimension, NetEase Cloud Music, NetEase Yanxuan and many other companies and products. technical solutions.
02. Three core R&D systems, the huge market has spawned the profession of virtual human maker
From a technical point of view, the birth of the virtual digital human is a combination of computer graphics, speech technology, deep learning, brain-inspired science, biotechnology, computing science and other technologies. People endow it with similar human appearance, behavior, values and other characteristics. What’s more important is its interactivity.
At this stage, the Fuxi virtual human team is about dozens of people, and technical personnel account for 75%. For NetEase Fuxi, the core R&D functional system of its virtual human business is divided into three parts:
- The first is the research and development of graphics algorithms, which builds the personalized ability of virtual humans to intelligently pinch faces and create images;
- The second is the research and development of animation-driven algorithms, which endows the virtual human with the driving ability of real-time smooth interaction and vivid migration expression;
- The third is the research and development of visual and rendering effects. In addition to improving the final presentation quality of the virtual human, it also ensures the high compatibility of Fuxi virtual human technology on commercial engines and other platforms.
“We believe that any technological innovation cannot be divorced from the actual business needs, and we have been reviewing some research and development topics that the team is doing.” Li Feng, the technical director of NetEase Fuxi virtual human technology, told 36氪 that the current team is focusing on It is how to synergistically improve the research and development of graphics and animation algorithms and the performance of visual effects rendering to achieve an expressive and real-time interactive virtual human effect.
So, how is a virtual human character born? In fact, the booming virtual digital human industry has spawned a new profession—virtual human manufacturer. This is not a single type of job, because virtual human manufacturing is a process work, including character determination, original painting design, design model, and algorithm. Research and development, effect joint debugging, real-time rendering optimization and other links, so it requires the cooperation of multiple professionals such as graphics engineers, character content planning, 3D art and so on.
Take Netease Yanxuan’s spokesperson “Lin Yaoyao” as an example. It is a virtual digital human made by Fuxi’s virtual human team based on a real-time rendering engine. In the early stage of production, the Fuxi virtual human maker team conducted a lot of research. In view of the differences in preferences between boys and girls, the team positioned Lin Yaoyao as the image of a sweet and cool millennial girl, which not only neutralized the preferences of different genders, It is also more suitable for spokespersons and other occasions, to facilitate the display of its vivid AI algorithm effects.
CG model Lin Yaoyao, using Fuxi AI expression transfer algorithm effect
Different from the production process of film and television animation, Lin Yaoyao’s production process relies on many Fuxi virtual human AI algorithms for efficiency and acceleration. For example, timbre conversion technology can convert the voice of any real person into a target timbre, and retain the content, mood and rhythm of the original voice, which enables Lin Yaoyao to not rely on a stable Zhongzhiren (from the Japanese “中の人” , refers to the person who manipulates the virtual anchor for live broadcast), and freely conducts real-time interaction such as live broadcast.
In addition, Lin Yaoyao’s production process also uses technologies such as ambient light acquisition and in-engine reconstruction, cloth calculation, and physical material rendering, so that it can better interact with the real world.
The end of the production process does not mean that the work of the virtual human maker has come to an end. The next step will involve planning work, which is mainly divided into two categories. One is content planning, which is responsible for packaging Lin Yaoyao, facing toC and toB. The overall design of the content, such as short video creative planning; the second is product planning, we need to consider how to productize and tool the entire technology chain behind Lin Yaoyao to better help downstream customers solve business problems.
03. In the fierce market competition, the differentiated advantages of Fuxi virtual human
Although the virtual digital human is not an emerging industry, the relevant report data shows that in 2020, the market size of my country’s virtual digital human industry will be about 200 billion yuan. The size of the digital human market may reach 270 billion yuan.
It is precisely because of this huge blue ocean market that a large number of players flock to enter. Whether it is an Internet technology giant or a new player, they all want to take a share of the virtual digital human market.
From the perspective of the player structure, there are currently big companies such as Microsoft, NetEase, Baidu, Tencent, ByteDance, iFLYTEK, Sogou providing rich solutions and AI technical support, as well as Autodesk, China Literature Group, Dayu, Plan Companies and startups such as Ren Intelligence and The Force have subdivided track layouts for content, tools, and CG, and output products at a single point.
In this fierce industry competition, what is the differentiated competitive advantage of Fuxi virtual human? According to Lao Yang, the head of product, compared with its peers, the differentiation of NetEase Fuxi virtual human is mainly concentrated in two aspects: product and business scenario.
The first is the product level. Fuxi virtual human takes 3D virtual human as the main type, and pays more attention to the algorithmic effect and real-time rendering performance of virtual human, rather than the pursuit of extreme lightweight. For example, for the 3D expression synthesis algorithm for virtual human, it is not easy to convert voice and text into animation of virtual human, and the connection is smooth, natural and delicate.
Therefore, the team will pay special attention to the impact of changes in human emotional state, speech rate, volume, etc. on mouth shape and facial expressions when speaking, and watch a large number of film and television performances to learn the performance habits of professional actors. Through the study of deep learning, the “expression animation parameter generation technology” researched by the team has been successfully applied to scenarios such as game plot animation production and virtual assistants.
Regarding the expressiveness of virtual human expressions, the team continues to think about how to further optimize, such as using machine learning to determine the naturalness of synthetic expressions. The algorithm expert of Fuxi virtual human innovatively proposed the research on expression coding. This work has won the double champion of AI top conference CVPR2022 and ICCV2021 international facial expression competition for two consecutive years. In the future, expression coding will also be added to the virtual human expression-related In the algorithm, create a more natural and vivid expression effect.
For the visual rendering effect, the Fuxi virtual human team believes that the future virtual human application must be real-time rendering (real-time interaction), high rendering quality (high immersion), so the team’s research and development are based on real-time rendering engine, and go deep into In the material reconstruction algorithm, by fully referring to the real hair and cloth material parameters, the fidelity of the virtual human rendering effect is greatly improved.
Secondly, at the level of business scenarios, Fuxi virtual human focuses on the fields of games and entertainment that he is familiar with, and conducts vertical explorations in these fields. At the same time, it is actively open, and conducts in-depth cooperation with B-end manufacturers who are deeply involved in other industries, and provides virtual human technology empowerment in its well-known application scenarios.
Empower virtual live broadcast manufacturers to create Vtuber live broadcast tools
04. Under the tuyere of the Metaverse, Fuxi’s virtual human’s future-oriented beachhead tactics
It is undeniable that the outbreak of the concept of “Metaverse” in the past two years has further stimulated the development of the virtual digital human industry. Many people regard virtual digital human as a new outlet for entering the Metaverse track. However, as an interdisciplinary field that spans multiple disciplines, virtual humans need to achieve a series of technological breakthroughs in order to truly face the future Metaverse and enter people’s life in the Metaverse.
“At this stage, virtual humans appear in various scenes in different forms, such as online video, offline interaction, etc., and different scenes focus on different effects.” Gale explained that, for example, video content currently pays more attention to the overall plot design. , visual quality, interactive applications pay more attention to the real-time and efficient interaction. In the face of the outlet of the Metaverse, there will be more content requirements for virtual people with more variety and effects requiring more integration in the future.
Next, the Fuxi virtual human team will focus more on real-time interactive virtual content to further improve the quality of real-time interactive effects and expand the business ecosystem. Among them, for the user to create a avatar in the virtual world, the team will continue to develop technologies such as high-degree-of-freedom pinch, high-quality & low-latency tone conversion based on the Mesh structure, so that C-end users can level, creating a virtual identity with a higher degree of freedom and better expressiveness.
The intelligent face pinching system developed by NetEase Fuxi allows users to create a virtual avatar by uploading a photo
In terms of business model, in order to speed up ecological expansion and cooperation, support the company’s existing 3D digital assets, or support the import of UGC assets, the Fuxi virtual human team will also launch automatic binding, automatic redirection and other capabilities, and public Fuxi virtual human art standards , to improve the docking efficiency of various avatars.
From the application point of view, the virtual human landing scene covers a wide range of fields such as toB, toC, and toG, but no matter which track it faces, the core competition of the entire market is always the industrialization and maturity of the production link of the entire technical tool process. An important guarantee of fast product delivery and reliability.
In the future, NetEase Fuxi will focus on the 2B business model and continue to pay attention to the development of virtual human standardized technical solutions. “We hope to actively expand industry ecological partners, carry out in-depth cooperation, and jointly create virtual human applications and achieve win-win cooperation.” Lao Yang said.
Posted by:CoinYuppie，Reprinted with attribution to:https://coinyuppie.com/netease-fuxi-virtual-human-commercialization-started-in-the-fourth-year-of-establishment-and-the-game-to-b-business-model-rushed-into-the-metaverse/
Coinyuppie is an open information publishing platform, all information provided is not related to the views and positions of coinyuppie, and does not constitute any investment and financial advice. Users are expected to carefully screen and prevent risks.