Armed with the express aim of enhancing visual quality and framerate in games, NVIDIA launched the first series of RTX graphics cards in 2018. Five years on, NVIDIA’s fledgling vision now embodies mass market appeal. Believe it or not, the RTX gaming ecosystem touches 500 titles. To this day, through many subsequent iterations and improvements, hybrid ray tracing and DLSS remain the two main thrusts baked into each RTX card.
A trip down memory lane is instructive insofar as it delineates the long and winding journey of GeForce RTX from a nascent vision to, now, millions of cards in the hands of gamers. September 20, 2018, is a date firmly etched into the annals of computer graphics. GeForce RTX 2080 arrived on the scene armed with hither-to unseen technologies promising to elevate the gaming experience to the next level. Pregnant with promise, NVIDIA forged on with RTX adoption through myriad 20 Series GPUs for both desktop and laptop computers.
It’s fair to say GeForce RTX came to the fore with the Ampere-based 30 Series GPUs. Offering a meaningful upgrade over extant 20 Series GPUs bearing the same name – RTX 3080, in particular, launched as a standout performer at the $699 mark – a greater number of RTX-imbued titles helped facilitate adoption. Fast forward to today, the RTX 40 Series rules the roost, and the champion GeForce RTX 4090 has no performance peer for either old-school rasterisation or forward-looking ray tracing.
Turn on and game
Now, RTX hardware and technologies have become almost synonymous. The company’s own numbers suggest that 83% of RTX 40 Series owners turn ray tracing on, while 79% have DLSS active. A great time, then, to examine how far NVIDIA has come in the pursuit of beautiful pixels with the latest updates on both fronts.
A deeper understanding of ray tracing and DLSS is important to contextualise multi-year focus. At its core, ray tracing comfortably beats traditional rendering techniques for the difficult but important task of generating accurate lighting in games and applications. GeForce RTX cards use dedicated hardware known as RT cores to accelerate complex calculations that model how a handful of casted light rays interact with pixels on the screen. The movement through material surfaces, reflections, and refractions informs the final pixel colours, and this is what leads to lifelike gaming. It’s not rocket science, but it’s pretty close.
Computer ray tracing therefore does a great job of making gameplay look more authentic by mimicking how the human eye perceives lighting. Helping generating photo-realistic imagery, popular Hollywood animation films also use this technique. The big difference in the gaming world is an RTX graphics card computes all necessary lighting in real time and for many frames per second.
Solving difficult problems
Nevertheless, it’s important to understand that various forms of ray tracing are so complex that it’s currently impossible to generate 60 frames per second at high resolutions. Even mighty cards such as the GeForce RTX 4080 and RTX 4090 struggle with the immense load, particularly at 4K. Generally running in concert with ray tracing, DLSS technology boosts framerate in a novel way.
Arriving at the same time as ray tracing, Deep Learning Super Sampling (DLSS) is a complimentary technology running on additional specialised hardware known as Tensor cores contained within every RTX graphics card. Understanding the constituent parts of the initialism offers important insight into how it works. Through artificial intelligence, DLSS improves performance by cleverly upscaling lower resolution frames to the native resolution you’re playing at.
Sounds easy, doesn’t it, though it’s anything but. Generating accurate extra detail from low-resolution inputs is no mean feat, and it’s right here that AI comes out to play. NVIDIA accomplishes the task by taking thousands of ultra-high-quality screenshots from a game engine. These images represent the absolute gold standard, or ground truth, but are impossible to run on a single graphics card at any semblance of decent framerate.
Spanning multiple servers in the cloud, a massive AI neural network compares ground truth images against various lower-resolution inputs used by DLSS and, through many millions of weighted operations, learns how to make informed choices about how best to add detail to them such that they remain as faithful as possible to the ground truth standard.
DLSS to the framerate rescue
NVIDIA generates a fine-tuned DLSS model through the process of exhaustive training. Once complete, the driver loads the model to an RTX card’s Tensor cores and is run locally. It is often tweaked over time as new DLSS standards and techniques come to the fore. The intrinsic beauty is that the lower load introduction by DLSS enables the graphics card to churn out more frames than is ordinarily the case, hence the performance boost you see.
First seen on graphics cards, this large-scale training and inference technology has broadened into generative AI used by applications such as ChatGPT. NVIDIA hardware runs many of these large-language models, by the way.
Two major recent releases known as DLSS 3 and DLSS 3.5 highlight the ever-evolving nature of the technology. Delivered in September 2022, DLSS 3 introduces Frame Generation technology that further boosts performance in a clever way. Only available on GeForce RTX 40 Series GPUs for now, entirely new frames are generated and inserted into gameplay. This is the next big step up from adding additional pixel data via regular DLSS.
Looking at performance with a GeForce RTX 4080 in Flight Simulator run at 4K, older generations of DLSS aren’t able to increase framerate significantly above native rendering. Frame Generation, however, ushers in a whole new league of performance for every setting. You’re looking at almost twice the framerate for DLSS Quality and Frame Generation over no technologies being active.
On its own, DLSS training is not smart enough to accurately generate entirely new frames that bypass the normal render engine. Only GeForce RTX 40 Series cards have a feature known as an Optical Flow Accelerator (OFA) that analyses two sequential frames. It’s able to determine the speed and direction of pixel-level information such as particles, reflections, shadows, and lighting. Used alongside motion vector data gleaned from the game, Frame Generation builds and inserts an entirely new frame at very little computational cost.
From a speed-up point of view, the best-case scenario is one where DLSS fills in most of the upscaling details on the first and third frames. Frame Generation creates and inserts a second frame that never existed before. With 4x DLSS upscaling and Frame Generation active, the game only needs to fully render 1/8th of the pixels. This approach is particularly useful if the geometry-producing CPU is proving to be a bottleneck.
There are caveats to this approach, you understand. The first is obvious insofar as Frame Generation runs only RTX 40 Series hardware. The second is that the game developer needs to have explicit support for the right specification of DLSS. With 50 games currently compatible and many more on the way, support for leading games engines such as Frostbite, Unreal Engines 4 and 5, and Unity is key to ongoing implementation success. If you don’t build it, they won’t come.
Ray-tracing optimisations
Plenty of innovation on the DLSS front puts ray tracing back in the spotlight. Any gaming frame has but a handful of casted rays as there’s not enough computational power to shoot them for every pixel. This is as true for consumer graphics cards as it is for Hollywood studios running offline computation on huge server farms. Running with this limited-ray approach, the resulting output is somewhat incomplete and noisy as there are parts of the scene where fewer rays intersect pixels. In fact, there are areas where rays are conspicuous by their absence.
This is where hand-tuned denoisers earn their keep. Their job is to generate the best possible quality by turning the noisier initial image into the smoother final output you see. This is done by accumulating pixels across many frames and blending appropriately, giving a cleaner approximation of each pixel’s colour in areas where rays can’t be cast.
However, the end result is not always optimal. The output is oftentimes distorted by the denoiser algorithm’s sub-par performance, particularly for complex lighting such as global illumination. There’s also the potential issue of ghosting, which is another by-product of inaccurate optimisation. Fidelity issues can occur because the denoiser’s temporal and spatial calculations aren’t, and can’t be, perfect in filling in the missing lighting data. The real-world impact can be profound, too. Imagine a racing game where the car’s headlight doesn’t seem quite right or instances where shadows are too soft and murky.
Reflections can also look poorer as the hand-tuned denoiser blends information, oftentimes muddying the visual waters. Making matters worse, the upscaling required for DLSS makes any visual anomalies all the apparent and worse.
Answering this clarion call on ray-traced quality is the latest improvement to DLSS known as 3.5.
Ray Reconstruction
Available on all RTX GPUs, the most notable feature of DLSS 3.5 is Ray Reconstruction, and it seeks to replace hand-tuned denoising with AI-generated smarts. Just as regular DLSS fills in the missing gaps to upscale images, Ray Reconstruction does effectively the same thing for ray tracing. It’s literally trained to understand the limitations imposed by hand-tuned denoisers. The underlying idea is to retain all the important information across multiple frames and use it wisely to generate better lighting effects. Processed on huge offline models that are up to 5x larger than DLSS 3, it works best on RT-heavy scenes that call the quality of lighting and reflections into sharper focus.
Once downloaded on to your graphics card through a driver update, this technical wizardry runs in the background. Completely oblivious to the user, though requiring both driver and title support, DLSS 3.5 is another genuine value-add inviting gamers to remain on the RTX ecosystem.
To be clear, this different approach to adding correct lighting into a ray-traced scene is subtle but important. Moving from hand-tuning to a large-parameter AI model serves up a far cleaner output that, according to NVIDIA, is similar in quality to the native image without ray tracing.
Furthermore, compatible with Super Resolution and Frame Generation, Ray Reconstruction doesn’t have a negative impact on overall performance because it replaces multiple hand-tuned denoisers with an AI-trained model. Sometimes it can provide a small performance boost for this very reason, as one algorithm replaces many.
Peering into Alan Wake 2
Theory is one thing, practise is another. Let’s examine DLSS 3.5 claims by looking at image quality in the well-reviewed Alan Wake 2. Along with Cyberpunk 2077: Phantom Liberty and Portal RTX, it’s a poster child for the new technology.
Built on the forward-looking Northlight Engine, Alan Wake 2 landed in October 2023 to popular acclaim, generating an average score of 89/100 on Metacritic. The liberal use of nighttime scenes, rain, and reflections sets an appropriately moody tone that’s right up DLSS 3.5’s street.
Running on a GeForce RTX 4090 at 4K, I used high-quality settings with three lighting options: ray tracing turned off, ray tracing set to high, and Ray Reconstruction turned on. Click on the above pictures and cycle through the trio. The immediate benefit of ray tracing is obvious and manifestly clear. Changes between Ray Reconstruction either off or on are more subtle, for sure, but still evident. The effect, actually, is more pronounced in real time than in screengrabs.
Toggling Ray Reconstruction results in the neon reflections becoming cleaner, with individual letters more easily discernible when it’s active. Also note the yellow reflections on the bottom right, instigated by a neon sign that’s not in view. The individual dots of the letter ‘R’ are definitely cleaner and more apparent than with it off. Furthermore, there’s greater clarity for other illegible letters.
Go back and look at the reflections by the tip of Casey’s gun. Turning Ray Reconstruction on results in slightly cleaner letters. The visual difference is purely down to the quality of a large AI-trained denoiser versus multiple hand-tuned ones.
Moving a bit further around, look at the diner sign’s reflection on the car’s bonnet. It’s blurry and barely intelligible without Ray Reconstruction. Immediately clearer once turned on, the scene feels more natural. Also note the cleaner reflection on the far part of the windscreen. If gaming is attention to detail, Ray Reconstruction certainly helps… at no cost to framerate.
Do appreciate that regular scenes don’t show stark improvements. Nevertheless, there appear to be no obvious downsides to its implementation, and it’s good to see that Ray Reconstruction is available on all GeForce RTX GPUs. In fact, after playing through the game for a few hours, it’s a no-brainer to keep it on.
You don’t even need to enlarge the pictures to show the clear improvements from Ray Reconstruction.
NVIDIA’s in-house video shows the manifest improvements when running DLSS 3.5 and ray tracing on. I know which one I’d rather game on.
Conclusion
Since the dawn of RTX GPUs NVIDIA has been on a mission to improve pixel quality and performance. Ray tracing and DLSS are the two cornerstone technologies on which the RTX platform now touches 500 games and applications.
DLSS has received more attention of late, particularly with a recent update incorporating the novel Frame Generation technique. Ray tracing, meanwhile, is more concerned with image quality, but both go hand in hand in pushing the visual envelope.
When done well, ray tracing impact is immediate and eye opening, especially for accurate lighting that lends a greater sense of theatre and realism to a game. That said, it’s not perfect, and never can be for real-time computer graphics. NVIDIA’s newest consumer technology is Ray Reconstruction, part of DLSS 3.5, and it seeks to address the issue of sub-par IQ caused introduced by the necessity of hand-tuned denoising.
Proving another feather in AI’s cap, Ray Reconstruction does a fine job of tidying the ray-traced output emanating from a limited number of casted rays. The sign of a good technology is that, once available, you never turn it off, and that’s how I feel about Ray Reconstruction in Alan Wake 2.
Broadening the church, Ray Reconstruction’s AI provenance means it runs on all RTX GPUs, which is a real plus in my book. Much like the first versions of DLSS, the technology is currently naturally stymied by limited adoption, but going by general RTX progress, there’s hope it will be rolled into many more games in 2024.
NVIDIA has built an enviable list of RTX-supporting technologies that combine to encourage gamers to opt or remain on the GeForce side of the fence. Available as part of DLSS 3.5, it’s safe to say Ray Reconstruction is one of the most innovative and interesting yet.
