Intel actually came to engage in GTA5 beautification MOD ……?

GTA6 is not yet released, is it because RStar used a similar ultra-frontline technology, the current host performance is not yet possible?

Recently a player made such a terrier chart: PS2 platform has three “Grand Theft Auto” (GTA), and GTA5 actually hard to carry the three generations of PlayStation platform ……

Intel actually came to engage in GTA5 beautification MOD ......?

(Actually, to be precise, there are a total of five GTA games on the PS2 platform, including three positive biographies and two biographies.)
The spoof is not malicious, do not get angry. This stalk chart can spread, but also because GTA5 is really the evergreen in the field of video games: since its release in 2013 in PS3/Xbox360 platform, including the total sales of single-player and online mode DLC has exceeded 140 million copies ……

This side of the development company R Star delayed the release of new works, the computer game platform CPU / graphics card hardware and display technology has been updated for several generations. Therefore, GTA5 beautification MOD has also become the direction of modification players are keen on. From realistic car models to a wider range of map and visual effects modifications, the variety of MODs has significantly enhanced the vitality of GTA5, making the game, which is almost 10 years old, still feel new to players.

Mod development is mostly done by foreign gods, and because of the gray area of violating user agreements, mod development is usually done by individuals on a non-profit basis. But recently, we were surprised to find that Intel is also “officially” developing GTA5 aesthetic MODs!

↑ May be slightly title party …… the actual situation is: Intel Labs spent years developing EPE (Enhancing Photorealisim Enhancement), a synthetic image fidelity algorithm based on deep learning convolutional neural network. It can optimize the original GTA5 output frame-by-frame, with a variety of third-party streetscape database, to generate different styles of realistic game graphics.

Intel Labs recently published a paper on their latest progress in deep learning image fidelity enhancement. Their experimental process uses images from GTA5. The Intel researchers designed a multi-module convolutional neural network architecture to learn from data from GTA5 and third-party city street view image libraries.

Intel actually came to engage in GTA5 beautification MOD ......?

The MODs made by the player gods rely mainly on prior swapping of graphics material, updating the rendering engine, adding light tracing support, etc.; Intel’s idea is to use the raw video and data output of the unmodified game directly, input it into the deep learning system, and act out the optimized image in real time.

Vladlen Koltun, the author of the paper, describes how this system works: it can be plugged into the game, and you can think of it as the image post-processing system for GTA5.

Intel actually came to engage in GTA5 beautification MOD ......?

(The left is the original GTA5 image, the right is after beautification)

Intel actually came to engage in GTA5 beautification MOD ......?

Action figure.

Intel actually came to engage in GTA5 beautification MOD ......?
Intel actually came to engage in GTA5 beautification MOD ......?

Koltun, one of the authors of the paper, is the chief scientist of Intel’s Intelligent Systems Division. He revealed at Eurographics 2021, the European graphics conference, that the EPE algorithm took two years to develop in Intel’s labs and that the results had never before been made public externally.

“(EPE) is the kind of research that takes a long time, doesn’t produce results for a few months, and is measured in years,” Koltun said, adding that “when it comes to (image synthesis fidelity), it’s not hard to get a little bit of results that you can show. It’s not hard to get a few good photos and put them in a paper; it’s very hard to invent a solution (something that really, really works) that really works.”

(It’s hard work, but at least the author of the paper has a reason to fish and play GTA5 during work hours)

Intel actually came to engage in GTA5 beautification MOD ......?

Vladlen Koltun talks about how the EPE algorithm works at Eurographics Conference
Let’s take a deeper look at how this “GTA5 Beautification MOD” from Intel Labs works.

Old “game”, new “gameplay”
GTA5 is almost 10 years old, but in both standalone and online modes, players have always invented interesting or weird new ways to play ……

Similarly, screen beautification MOD is also a field that has existed with GTA5 for many years, and there are only a few “folk” technical directions. This time, Intel Labs has used a new “game” to achieve an unprecedented breakthrough in graphics fidelity.

The training principle of EPE algorithm is as follows.

Intel actually came to engage in GTA5 beautification MOD ......?

In short, EPE integrates three separate neural networks that process three different data sources: the buffer data output by GTA5’s own rendering engine, the game’s straight-out screen, and images from a third-party street view database.

(1) GTA5 game comes with a rendering engine that can output a set of data called G-buffer, which includes the type of objects in the screen, the distance between the player’s view camera, surface material, smoothness, albedo, lighting data and so on.

Intel actually came to engage in GTA5 beautification MOD ......?

These data, fed into a set of convolutional neural network data streams, extract the mask maps of various types of data for training the neural network; then, after a series of specially designed residual modules, tensor feature data of different scales are output.

Intel actually came to engage in GTA5 beautification MOD ......?

2) With the training result of the previous step, train an image enhancement neural network, and then input the game straight out of the screen into this neural network to get the enhanced image.

Intel actually came to engage in GTA5 beautification MOD ......?

3) The real image translation work happens in this step: the game straight out, enhanced images, and similar images from the third-party street view database are fed into a perceptual discriminator (neural network), and after a series of calculations, the perceptual feature map, label map, and related interpolation data are generated, and finally “translated” into a The final “translation” is a graph.

This part also adopts the design of the adversarial model, after continuous training, the final output image can accurately restore the composition of the objects in the game straight out of the screen, and perfectly transfer the style of external database images.

For example, the image below shows the style of the image generated by Intel Labs using the combination of GTA5 and Cityscapes database. Since most of the Cityscapes image data comes from Germany, which has a unique image style, you can see that the “beautified” result seems to be greenish and somewhat gloomy.

Intel actually came to engage in GTA5 beautification MOD ......?

And, because of the more lush German vegetation, you can see the bald, dry, yellowish hills of San Andreas (based on Los Angeles) in GTA5 rendered in dense green vegetation.

Technological innovation
As Silicon Starman understands it, there are several innovations in the image translation approach proposed in this paper.

As mentioned before, traditional ideas for implementing GTA5 aesthetic mods require heavy modifications to the game files, risking file corruption, and are difficult to use in multiplayer online mode, as well as involving gray areas that violate the game’s usage agreement.

The difference between the EPE idea is that it is directly based on the game output screen and does not involve modification of the game files, so it does not violate the usage agreement. (Of course, as mentioned earlier, it does need to “listen” to the rendering engine output while the game system is running during training.

Of course, EPE is not the first technology to use deep learning ideas for GTA5 graphics optimization.

Before that, UC Berkeley and Adobe Research developed CUT (Contrastive Unpaired Translation), and Nanyang Technological University, UCB, and Shang Tang developed TSIT (Two-Stream Image-to-image Translation) – both of which were previously the world’s leading image translation algorithms.

Intel actually came to engage in GTA5 beautification MOD ......?
Intel actually came to engage in GTA5 beautification MOD ......?

However, both algorithms suffer from severe image distortion, at least in GTA5. For example, the CUT renders small objects that are relatively independent from their surroundings (e.g. trees, signs, pedestrians, etc.) with ghosting and poor timing stability.

Intel actually came to engage in GTA5 beautification MOD ......?

The TSIT algorithm rendering results in severe distortion in the upper sky area of the screen, adding some non-existent vegetation.

Intel actually came to engage in GTA5 beautification MOD ......?

Intel Labs scientists believe that these algorithm distortions may be caused by the third-party database used for training. For example, the angle of the camera is too low causing trees in the distant view to fill the upper part of the screen. As shown in the image below, many of the photos in the Cityscapes database have large areas of vegetation, while usually the vegetation in the GTA5 screen is small.

Intel actually came to engage in GTA5 beautification MOD ......?
Intel actually came to engage in GTA5 beautification MOD ......?

Even in the Cityscapes database, because there is a Mercedes-Benz decal on the data collection vehicle, other algorithms mistakenly think that this decal is also part of the road feature when rendering.

Intel actually came to engage in GTA5 beautification MOD ......?

When training EPE, Intel’s team deliberately narrowed the neural network’s field of view so that it could focus on specific objects in the frame, and EPE achieved the most advanced level of realism and temporal stability in frame-by-frame beautification of GTA5 images, significantly outperforming deep learning-based image-to-image translation algorithms such as CUT and TSIT.

Because of the design of the algorithm, it also has a bit of a “plug-and-play” feel, and can access a variety of external databases to achieve different styles of beautification results.

For example, Mapillary Vistas, a database of street images from around the world, is more diverse and colorful. This style can also be translated into the GTA5 screen by EPE algorithm, the effect is more like the impression of San Andreas.

Intel actually came to engage in GTA5 beautification MOD ......?

On the left side of the picture below is the straight out of the game, and on the right side is the result of the beautification using Mapillary Vistas as the target style, you can see that the color saturation of the picture has been greatly improved; the fog added to the distant scene in order to reduce the pressure of the game rendering has also been optimized.

Intel actually came to engage in GTA5 beautification MOD ......?

Netizen comment: What? This is actually not true?

On May 11, Intel Labs posted the EPE introduction video to YouTube. After only half a month, this video already has more than 2.7 million views ……

Most of the netizen comments are from a layman’s point of view, but they all express shock at the rendering results demonstrated by the team ……

Fat Tabby commented: If you show me this video and tell me it’s from a car recorder, I should believe you.

Intel actually came to engage in GTA5 beautification MOD ......?

G Luong said: This is the real image simulation. All the other screen beautification MODs are just adding reflections and making every road very wet.

Intel actually came to engage in GTA5 beautification MOD ......?

There are also several sharp-eyed players found that if you follow the Cityscapes style of beautification, out of the picture style is actually with GTA4 (set in Liberty City, New York as the prototype, the style is darker).

Intel actually came to engage in GTA5 beautification MOD ......?
Intel actually came to engage in GTA5 beautification MOD ......?

Netizen OK DOK said, no wonder when GTA4 came out, people were surprised that a game can be so “real”.

Intel actually came to engage in GTA5 beautification MOD ......?

Netizen Cosine is right: not so cool reflections, no oversaturated colors and ultra-high brightness, is the most “real” game.

Intel actually came to engage in GTA5 beautification MOD ......?

Interestingly enough, Intel and GTA5 are already old friends – not at the game level, of course, but at the research level.

Back in 2016, Intel Labs and the Technical University of Darmstadt in Germany were trying to do research on autonomous driving with GTA5. At the time, autonomous driving had become a new technology trend, but not everyone had access to a lot of visual data from real roads for research, due to realistic costs and other factors.

So Intel and the university formed a team to try to extract data from games like GTA5 that approximated real road conditions to help train self-driving technology. They also published a paper at the time, Playing for Data: Ground Truth from Computer Games –

Among that team was Vladlen Koltun, who is featured in today’s EPE paper.

Intel actually came to engage in GTA5 beautification MOD ......?

So today we can say that the chief scientist of Intel’s Intelligent Systems Division is also a god in the field of GTA5 aesthetic mods!

Do you think that Koltun will be as anxious as everyone else, waiting for the release of GTA6 by R-Star?

Of course, if GTA6 can also use similar deep learning image simulation technology, Silicon Star people are still willing to wait a little longer ……

–On second thought, do you think the reason GTA6 has not been released yet is because RStar is using some super cutting-edge technology that current console performance – even PS5 – cannot achieve? -are not yet achievable?

Posted by:CoinYuppie,Reprinted with attribution to:https://coinyuppie.com/intel-actually-came-to-engage-in-gta5-beautification-mod/
Coinyuppie is an open information publishing platform, all information provided is not related to the views and positions of coinyuppie, and does not constitute any investment and financial advice. Users are expected to carefully screen and prevent risks.

Like (0)
Donate Buy me a coffee Buy me a coffee
Previous 2021-05-30 07:14
Next 2021-05-30 07:32

Related articles