NVIDIA is at the forefront of bringing the power of AI to consumers through our GPUs, and over the next few years we will see several new technologies emerge that will usher in a new era of gaming and graphical fidelity.
NVIDIA’s DLSS 3.5 and ray reconstruction are just the beginning, as the company talks about several next-generation AI techniques that will fundamentally change how consumer GPUs generate pixels
With tech giants like Microsoft and Amazon engaged in a race to integrate AI, the influx of artificial intelligence into every mainstream application known to the human world seems imminent due to the enormous benefits this technology offers, whether it be in the form of advanced computing or improved consumer experiences.
But when it comes to leveraging AI in computing and enterprise applications, NVIDIA is at the forefront, and a prime example of this is NVIDIA’s ACE. The company now seems determined to bring the power of AI to improved graphics performance. This article is not mainstream news, but rather a series of events that show how big a role AI will play in GPU graphics performance, ultimately opening new doors in how developers visualize their ideas “graphically”.
NVIDIA’s Ray Reconstruction: Combining RT and AI in a Single Package for Massive Visual Enhancements
In the modern era of graphics, the race is on architectures to upscale native resolutions and allow consumers to view best-case scenarios through “artificially enhanced” means such as NVIDIA’s DLSS and AMD’s FSR. NVIDIA has said that the influx of AI could potentially enhance the pixels users see from “seven to eight” and in some cases “quadruple” the resolution. This has allowed the company to allow developers to “recreate” older titles by leveraging the power of DLSS.
To provide a bit of background, NVIDIA’s previous ray tracing pipeline relied on several elements, from denoisers (used to remove noise artifacts) to anti-aliasing filters, to achieve the results seen in technologies like DLSS 2. While this served a purpose, it prevented developers from taking advantage of “image upscaling” because the pipeline simply couldn’t support it. However, there’s a lot of complexity packed into this statement.
So going forward, integrated image upscaling is what’s needed, and to achieve this, NVIDIA decided to introduce “Ray Reconstruction”. This particular mechanism changes the way the ray tracing pipeline works, adopting an “integrated denoiser” approach. Ray Reconstruction employs multiple AI models designed to handle highly dynamic signals such as moving shadows, light sources, and objects, and as a result, delivers significantly improved performance compared to hand-crafted denoisers.
Ray Reconstruction will be a major advancement in the realm of “AI-driven” graphics, optimizing the ray tracing process and making it accessible to all consumers, and potentially even removing hardware limitations. Not only that, but it will also remove the boundaries of graphics computation, allowing developers to achieve incredible visuals and even recreate classic titles to “graphically transition” them into the modern era.
NVIDIA’s Next Step in AI Graphics
But where does it go from there? NVIDIA’s John Burgess recently covered some of the emerging trends in AI graphics at the High Performance Graphics event, specifically talking about consumer-level GPUs like RTX for GeForce and RTX for Workstations. AI is said to be useful for a variety of rendering tasks beyond post-processing like DLSS. This is also hinted at by Intel’s TAP, which is looking at applications of AI beyond just upscaling and frame generation. An example of this was also shown by NVIDIA at the same event, which we’ll discuss later.
Some of the new approaches NVIDIA is proposing include:
NVIDIA Improves Caching and Performance with Neural Texture Compression, NeuralVDB and Neural Radiance Cache Models
The first example of leveraging AI to improve graphics fidelity is neural texture compression, which uses small MLPs (multilayer perceptrons), which are artificial neural networks made up of connected neurons. The neural texture compression model uses one MLP network per material texture stack and contains two hidden layers.
This model achieves 4-16x compression ratios compared to standard block compressed textures or BC, allowing higher resolution textures in the same memory footprint and comparable resolution in a much smaller memory footprint, allowing GPUs with limited VRAM, bandwidth and cache to process high resolution textures more efficiently.
Next is NeuralVDB, which represents compressed volumetric data and a sparse tree topology. It uses 2-4 MLPs and 3-4 hidden layers per volume, achieving 10-100x compression. At SIGGRAPH 2022, NVIDIA demonstrated how this model can be used to perform complex volumetric simulations while reducing the memory footprint by up to 100x.
Finally, there is the Neural Radiance Cache, which uses a neural network to encode radiance information. The model uses one MLP network per probe with two hidden layers, allowing probes to be dynamically updated (training) and queried (inference). Adding the Neural Radiance Cache to path tracing rendering significantly improves sample quality, which speeds up conversion and makes it easier to denoise rendering.
NVIDIA’s plans to enhance real-time rendering include the use of neural appearance models, which will bring about significant improvements.
Leaving ray tracing behind, let’s talk a bit about the fundamental building blocks of graphics computing: shading performance and real-time rendering. NVIDIA has proposed a very interesting way to organize this field. As we covered in a previous article, the company has taken advantage of the capabilities of neural material models, which we will discuss shortly. For now, Team Green is assembling neural networks that will allow us to boost graphics computing to new levels that were previously unimaginable.
During the SIGGRAPH 2024 keynote, NVIDIA unveiled Neural Appearance Models (NAMs), which leverage AI to represent and render the appearance of materials in a more highly optimized and realistic way than traditional methods. These neural models are trained to recognize the visual properties of real-world materials, and by applying such datasets to rendered images, they create a final product that is not only highly realistic, but also much faster.
Explaining the complexities of neural networks can be difficult, especially for a general reader, but I will try to summarize it in the most effective way here. The neural appearance model discussed earlier is built on certain blocks including two MLPs (Multi-Layer Perceptrons), one for BRDF estimation and the other for importance sampling and data sampling, but it is extended to a whole new level. In addition to that, NAM utilizes an encoder-decoder architecture, which processes the input data and generates the final material appearance based on the parameters along with the dataset.
Now that we know how NAM works, let’s get to the interesting part: its potential. NVIDIA’s introduction to its Neural Appearance Model revealed that it is capable of rendering texture resolutions up to 16K, which is a huge leap forward. Apart from this, the on-board computationally efficient neural network is said to reduce rendering times by as much as 12-24 times, which is also a huge achievement considering that this was previously thought impossible with traditional shading graph techniques.
Looking ahead, NVIDIA believes MLP, and AI in general, can enable major improvements in the world of graphics.
Simple MLPs can be surprisingly powerful if:
Data compression Complex mathematics approximations Complex signal data caching
Performance is on par with traditional rendering.
Layer Fusion: Exploiting Accuracy and Sparsity Reduction
Because MLPs are small, they can perform on par with traditional rendering, and these enhancements do not come at a significant cost.
assignment:
Divergence intermix with conventional shader cores
Some of the challenges include essentially the divergence that must be overcome if each thread in the GPU is querying/running its own neural network to get texel values, as these threads are supposed to work together. Thus, there is both execution divergence and data divergence.
An example of the future of AI rendering was shown in the form of a recent OpenAI video created using Sora. The video shows a jeep driving over rolling dirt terrain leaving a realistic dust trail behind, with the vehicle showing off realistic weight and simulation based on the terrain. This is an entirely AI-created video and offers a glimpse into future applications of AI, such as gaming. This little animation required tens of thousands of GPUs to train using a small text prompt, but as AI hardware becomes more powerful, it will likely make its way onto consumer GPUs in the coming years.
Something else that was interesting in this session was the comments regarding the use of dedicated hardware such as NPUs and GPUs, as well as other future accelerators that may be integrated into future GPUs.
The problem with dedicated hardware that’s not tied to the GPU is that you lose the whole ecosystem that you can use for whatever neural network stuff you’re using, so we prefer to have it pretty tightly coupled so that you can run programmable code in the SM, go to the tensor cores, and go back as many times as you need for your particular problem, so that someone can build their own dedicated hardware to just run the convolutional encoder and do whatever other stuff they need to do.
I don’t want to talk about future products or anything like that, but before the Neural Appearance Model was taking shape and starting to work, I spent a lot of time thinking about how to accelerate materials. For example, if you have a great Uber material, or Arnold shader, you can build dedicated hardware to handle something complex with 50 inputs and accelerate that, or sample multiple layers with Monte Carlo or position-free Monte Carlo. We basically didn’t get anywhere, but I don’t think the next obvious disruptive change is going to come from dedicated hardware. I think the next disruptive change is going to be using the tools that we enabled to take the next step without needing the hardware, because the hardware is already accelerating the building blocks to build something new and exciting. I think the Neural Appearance Model is a great example of that. We’ve already built the hardware, but we didn’t know if it was a good fit for it until we actually tried it.
John Burgess – NVIDIA
It is no exaggeration to say that the impact of AI on the world of computing is yet to be measured, given the limitless possibilities this technology has brought. Its impact on graphics computing is just a small sample of its potential capabilities. Neural appearance models and ray reconstruction will definitely bring next-generation graphics to a level we once dreamed of. This is the result of the efforts of NVIDIA and our team, not to mention the role that hardware power plays here.
