Fidelity Super Resolution in 2021

[PrincessFrosty]

A software based solution is always going to be fundamentally inferior to that of a hardware based approach.

People who think this is going to achieve DLSS 2.0 levels of performance ( & IQ levels) are in for a big disappointment, just as is the case with their weak hybrid software/hardware based Ray Tracing performance levels.

I second this. If I had to put money on this it's going to fall short of DLSS 1.0 which was fairly bad IMO. At this stage it feels it's more likely a project to get some parity with Nvidia on features on paper rather than produce any real value. If this could have been done well with general purpose compute functions it would have been made years ago.

 

[henson0115]

you do realise that the hardware will do the calculations provided by the software right

you also realise nvidia doesnt do the calculations on teh gpu but a server in a data center passed to you via a driver update.

 

[nvidiamd]

I don't even understand what software vs hardware means. You will probably need to install a program called FSR, then click on FSR.exe icon on the screen and open the game and choose resolution from inside FSR program. That is software based solution.
All of this from the myth that Nvidia has "tensor cores" while AMD does not have them.
Facts - if you strip 80% of an AMD CU, you will end up with a tensor core. The same if you strip 80% of a 1080 CU and use the rest to do the maths the "tensor cores" are doing.
Tensor cores are "specialized hardware" in the sense that they can only do this type of math, unlike a normal CU that can do a lot of things.

you do realise that the hardware will do the calculations provided by the software right

you also realise nvidia doesnt do the calculations on teh gpu but a server in a data center passed to you via a driver update.

Yeah we can say that DLSS is a software based solution.

 

[Nexus18]

Have a look into things like madvr (video renderer used to improve up/down scaling which performs better with the correct hardware i.e. gpus [and primarily nvidia gpus] as opposed to the CPUs doing the work) and also the likes of googles phone where they have a dedicated image processing chip to help speed up the processing of photos etc. and you'll see why having the "correct and dedicated" hardware always beats out a software approach.

Same way there is "hardware vs software" methods of encoding.

There really is a lot more going on in the background and it isn't a clear cut of "software vs hardware"

It would be good if nvidia clarified exactly the process of their dlss implementation though as it is rather vague:

DLSS uses the power of NVIDIA’s supercomputers to train and improve its AI model. The updated models are delivered to your GeForce RTX PC through Game Ready Drivers. Tensor Cores then use their teraflops of dedicated AI horsepower to run the DLSS AI network in real-time. This means you get the power of the DLSS supercomputer network to help you boost performance and resolution.

 

[nvidiamd]

Have a look into things like madvr (video renderer used to improve up/down scaling which performs better with the correct hardware i.e. gpus [and primarily nvidia gpus] as opposed to the CPUs doing the work) and also the likes of googles phone where they have a dedicated image processing chip to help speed up the processing of photos etc. and you'll see why having the "correct and dedicated" hardware always beats out a software approach.

Same way there is "hardware vs software" methods of encoding.

There really is a lot more going on in the background and it isn't a clear cut of "software vs hardware"

It would be good if nvidia clarified exactly the process of their dlss implementation though as it is rather vague:

You don't need to look at madvr, you can look at a game using software rendering, i think Linus tried to run Crysis on a Threadripper. And while it can be done, it won't run near as fast as it runs on a medium graphic card. That is the nature of software emulation. To play a PS3 game you will also need a computer much stronger than a PS3 if you run the game through software emulation.
You can probably emulate DLSS too if you have access to Nvidia code. There is no magic there any CPU can be programmed to do a lot of things.

The first hardware accelerated thing i remember was the MMX thing in Pentium. What was that? They used a part of their CPU to run a particular set of instructions. It was no different than the tensor cores and DLSS.
AMD last gen is also able to do "MMX" when needed. So at least on the 6000 series and on consoles, the FSR will be "hardware accelerated". No different than tensor cores and DLSS. I am not sure about the older series and the other devices/CPU's but i am sure the 6000 series is the same.
But wait Nvidia has "dedicated hardware" while AMD does not.
In fact Nvidia is using this design because it is common to their Pro line of cards. They could had simply do what AMD did, cut the tensor cores and use more CU's and use these when upscaling was needed, or use the whole power of more CU's when upscaling was no needed. But that means different chipsets for the Quadro and so lower profits.

 

[Aegis]

It would be good if nvidia clarified exactly the process of their dlss implementation though as it is rather vague:

So basically Nvidia trains DLSS back at Nvidia HQ using a vast amount of different games so that the AI 'knows' what video games look like - the tensor cores run the DLSS code and upscale low-res images using this training model to enhance the detail in the image. Any performance gain is because you're physically running at a lower resolution - it's not perfect but Nvidia are continually refining both the software (i.e. DLSS itself) and the training data (which is supposedly delivered in driver updates).

 

[nvidiamd]

So basically Nvidia trains DLSS back at Nvidia HQ using a vast amount of different games so that the AI 'knows' what video games look like - the tensor cores run the DLSS code and upscale low-res images using this training model to enhance the detail in the image. Any performance gain is because you're physically running at a lower resolution - it's not perfect but Nvidia are continually refining both the software (i.e. DLSS itself) and the training data (which is supposedly delivered in driver updates).

There is no vast amount of different games you are doing it for each game. You feed the computer with low res images and their higher res versions until the computer understands how to upscale pretty much everything in that game. Then you get a code based on "tensor" math. And you run that code on the tensor cores.

 

[ltron]

In the article it states "Finally, AMD says that the deep learning approach does not take into account the aspects of the original image, which may lead to lost color or details in the final image."

I could have sworn I noticed slight gamma shift/more washed out image in some games with DLSS, maybe this is the reason.

 

[ltron]

I second this. If I had to put money on this it's going to fall short of DLSS 1.0 which was fairly bad IMO. At this stage it feels it's more likely a project to get some parity with Nvidia on features on paper rather than produce any real value. If this could have been done well with general purpose compute functions it would have been made years ago.

Not necessarily, the comparatively simple Resizeable Bar could have been done years ago too and wasn't. Reading the PC Gamer article on the patent it sounds very impressive, this is not the bumbling AMD of 5-10 years ago. The sceptics may find themselves eating crow. https://www.pcgamer.com/uk/amd-gaming-super-resolution-patent/

 

[Aegis]

There is no vast amount of different games you are doing it for each game. You feed the computer with low res images and their higher res versions until the computer understands how to upscale pretty much everything in that game. Then you get a code based on "tensor" math. And you run that code on the tensor cores.

Supposedly DLSS 2 changed that - according to Nvidia "The original DLSS required training the AI network for each new game. DLSS 2.0 trains using non-game-specific content, delivering a generalized network that works across games. This means faster game integrations, and ultimately more DLSS games.”

https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-2-0-a-big-leap-in-ai-rendering/

 

[nvidiamd]

Supposedly DLSS 2 changed that - according to Nvidia "The original DLSS required training the AI network for each new game. DLSS 2.0 trains using non-game-specific content, delivering a generalized network that works across games. This means faster game integrations, and ultimately more DLSS games.”

https://www.nvidia.com/en-us/geforce/news/nvidia-dlss-2-0-a-big-leap-in-ai-rendering/

Honestly i have my doubts. It can work for games using UE4 or Unity as long as the devs don't use their own assets. But i think for most games it still needs some level of per game training if you want a good result.
Again there is no magic, the tensor cores are doing some matrix math that is named exactly tensor. RDNA2 CU's are also trained to hardware accelerate this type of math. The "AI" part is not inside your GPU, it is inside Nvidia and AMD servers.

 

[Aegis]

Just quoting Nvidia's documents

"Once the network is trained, NGX delivers the AI model to your GeForce RTX PC or laptop via Game Ready Drivers and OTA updates. With Turing’s Tensor Cores delivering up to 110 teraflops of dedicated AI horsepower, the DLSS network can be run in real-time simultaneously with an intensive 3D game. This simply wasn’t possible before Turing and Tensor Cores."

 

[nvidiamd]

Just quoting Nvidia's documents

"Once the network is trained, NGX delivers the AI model to your GeForce RTX PC or laptop via Game Ready Drivers and OTA updates. With Turing’s Tensor Cores delivering up to 110 teraflops of dedicated AI horsepower, the DLSS network can be run in real-time simultaneously with an intensive 3D game. This simply wasn’t possible before Turing and Tensor Cores."

I'll say lets wait for FSR and check to see the performance impact it has on the Nvidia 1000 series. If FSR is based on that patent that was shown today it is also using tensor maths. And if it will have any improvements on the 1000 series then the DLSS could had also be run on the 1000 series ( most likely not with the same results as it has on Turing or Ampere but still... ).
As about AI, check how the tensor math looks. Do you think the tensor cores are comparing images or are thinking how a bridge, a car or a house should look like? It's all in the matrix, the AI inside the card is just a marketing trick.

 

[Grim5]

The AMD patent leaked has nothing to do with normal Super Resolution that AMD has previously spoken about.

AMD specifically said in February it's super resolution would not be a AI based implementation, it's a open source software that runs on any graphics cards regardless of it's architecture.

But the patent is completely different to this, the patent is very similar to how DLSS works

This is what I believe because its a freakin no brainer.

* AMD in it's rush, came out and said we've got a competition to DLSS and it runs on everything, its super easy to implement at the driver level and developers dont need to do anything.

* Gamers frothed over themselves and laughed at DLSS

* Earlier this year, AMD in its testing found that FSR looks like absolute rubbish next to DLSS (this has been confirmed by various leaks).

* AMD immediately started working on a replacement that uses AI like DLSS and thus the patent was created. This replacement most likely will not run on RDNA2 but requires an RDNA3 GPU

 

[nvidiamd]

* AMD immediately started working on a replacement that uses AI like DLSS and thus the patent was created. This replacement most likely will not run on RDNA2 but requires an RDNA3 GPU

The patent is older than RDNA2.

 

[Gerard]

That'll be your 720p 290x

Nope, that would be the 3070FE i had from December until a few weeks ago

 

[Gerard]

* AMD in it's rush, came out and said we've got a competition to DLSS and it runs on everything, its super easy to implement at the driver level and developers dont need to do anything.

* Gamers frothed over themselves and laughed at DLSS

* Earlier this year, AMD in its testing found that FSR looks like absolute rubbish next to DLSS (this has been confirmed by various leaks).

* AMD immediately started working on a replacement that uses AI like DLSS and thus the patent was created. This replacement most likely will not run on RDNA2 but requires an RDNA3 GPU

Oh here we go, pray tell what "various leaks" would these be? If you're on about the Metro Exodus interview then they already confirmed they were talking about the fidelity fx toolkit and not fsr. You'd think you would be clued in enough to take "leaks" with a truckload of salt as more often than not it's some random dingleberry spouting a load of crap 50/50 in each direction for youtube clicks.

 

[PrincessFrosty]

Not necessarily, the comparatively simple Resizeable Bar could have been done years ago too and wasn't. Reading the PC Gamer article on the patent it sounds very impressive, this is not the bumbling AMD of 5-10 years ago. The sceptics may find themselves eating crow. https://www.pcgamer.com/uk/amd-gaming-super-resolution-patent/

I think resizable bar has numerous hardware compatibility problems and probably only started to exist because some vendors were had a hand in making each individual component required and so could force the coordination to make this work. When I make that comparison I'm kinda talking about something that is software only had has no real barrier to entry, other than maybe performance constraints, and even then for a feature that's supposed to save performance if it could have been done in software only, you would have expected this to have been a priority in the past.

The AMD patent leaked has nothing to do with normal Super Resolution that AMD has previously spoken about.

AMD specifically said in February it's super resolution would not be a AI based implementation, it's a open source software that runs on any graphics cards regardless of it's architecture.

Reading the linked article and I think you're right, they are taking a AI implementation which at the very least would have bad execution times if not done on fixed function hardware like tensor cores or and equivalent. So something doesn't quite add up there.

Of course I'll happily eat crow on this one if I'm wrong, but the principles at play here don't make much sense to me. There's really a software implementation out there which is going to improve image quality and lower performance demands and it's all a straight up win, with no drawbacks. And it's just been conceptually hidden from being discovered all this time? I don't really buy that, I think one of these promises will fall through and my gut feeling is it'll be in image quality of the final output. I suspect it'll be something like a more advanced version of an image upscaler that you'd find in photoshop, possibly temporal, only with tricks for fast execution and probably some tricks to mask edge case artefacts. I'll also bet it finds most of its use on the consoles where the users have no real knowledge or appreciation of image quality and you can get away with rough and ready tricks and no one really knows or cares.

 

[Woodsta888]

Nope, that would be the 3070FE i had from December until a few weeks ago

I knew - I was messin'

 

[PrincessFrosty]

There is no vast amount of different games you are doing it for each game. You feed the computer with low res images and their higher res versions until the computer understands how to upscale pretty much everything in that game. Then you get a code based on "tensor" math. And you run that code on the tensor cores.

For DLSS v1.0 this was true, but DLSS v2.0 moved to a generic model which is trained on a variety of different games and has generic implementation, there's no more game specific training or game specific implementation.