Microstutter - fact or fiction? Let's find out once and for all...

[Pulseammo]

After a quick scan of the thread I think I understand the VSYNC issue in relation to microstutter:

Say you have a display going at 60hz, your graphics card will want to do 60fps. What if it cant do 60fps? Well, 55fps is no good because it will cause tearing, which is what VSYNC is trying to avoid. So it has to drop to the next framerate that it can handle but still be in synch, which I think is 30fps. So because you drop from 60 -> 30 and back up to 60, you've effectively "microstuttered".

So if you take a game like HL2, it's easier for your card to output 60fps all the time. I dunno what GRID is like, but I'd suspect it's harder to pull off the frames required to stay at 60fps.

Please don't kill me if I'm wrong, but I think thats how it works!

 

[Dr_Fartypants]

After a quick scan of the thread I think I understand the VSYNC issue in relation to microstutter:

Say you have a display going at 60hz, your graphics card will want to do 60fps. What if it cant do 60fps? Well, 55fps is no good because it will cause tearing, which is what VSYNC is trying to avoid. So it has to drop to the next framerate that it can handle but still be in synch, which I think is 30fps. So because you drop from 60 -> 30 and back up to 60, you've effectively "microstuttered".

So if you take a game like HL2, it's easier for your card to output 60fps all the time. I dunno what GRID is like, but I'd suspect it's harder to pull off the frames required to stay at 60fps.

Please don't kill me if I'm wrong, but I think thats how it works!

It's a good point, and yes, with vsync the framerate does drop to 30fps (or another integer division of 60) when the framerate can't keep up. However, that's not what's happening in the GRID benchmark for two reasons:

1) I take into account the possibility of these kinds of effects (along with paging and other 'real' stutter effects) by 'culling' the worst 2% of frametime variations before taking the average to calculate the index. So, unless more than 2% of frames are next to 'jump' frames, the effect won't come out significantly on the result

2) The average framerate is pretty much bang on 60fps, implying that it has not dipped below 60fps during the benchmark (I use all frames to calculate the average fps).

On top of this, rounddodger sent me the benchmark file, and it's really all over the place. Quite strange, actually... I imagine it's something GRID specific.


Anyway, I don't want you to think I'm 'jumping down your throat' with this - it was a very good idea. It's nice to see people thinking about the 'how and why' rather than just the 'what'

 

[BubbySoup]

Just to be clear, are we still saying that microstutter is unique to multi-GPU setups, and that all the people on this thread claiming microstutter on single GPU setups are actually suffering from some other, unrelated performance hitch?

 

[DragonRR]

@Dr. Fartypants.

I too have scanned through this thread and read your text on your methods.

I am a little puzzled. You mention that you cull the worst 2% of frametime variations..

Microstuttering from my own experience and understanding happens when the second GPU fails to "hit" the next frame in time and so the output from the second GPU is ignored. This is effectively (imo) just like the problem with VSYNC when the frame rate drops below the monitor SYNC rate, the frame is ignored and you get a stutter.

It SEEMS from your descriptions that you are trying to ignore this type of stutter but instead are looking for general variations from the total average frame rate. To me this means that you are looking at normal drops in frame rate from the average which you would get with any GPU. If this is the case then I would be the first to say that the measurement isn't worthless but it doesn't imo measure microstuttering?

EDIT:
@BubbySoup. Yes. You can get stuttering in any setup from paging, texture loading, high poly scenes and so on. But from my experience microstuttering is slightly different, you get tiny freezes with multiGPU setups when the framerate falls lower than around 60fps.

 

[Dr_Fartypants]

Just to be clear, are we still saying that microstutter is unique to multi-GPU setups, and that all the people on this thread claiming microstutter on single GPU setups are actually suffering from some other, unrelated performance hitch?

There will always be some degree of 'microstutter', even with single GPUs. It is actually this irregularity of frame output which makes it impossible to predict when to output frames in multi-GPU AFR mode, and leads to 'noticable' microstutter! For the most part though, it seems small with single GPU setups (less than 10%).

So far, with the few results we have with multi-GPU setups, it seems that on some games we see much larger microstutter values (up to 40%), whereas on others, particularly ones that don't make proper use of x-fire, we see similar values to the single GPU setup.

This being said, I don't think we have enough SLI/x-fire results yet to start drawing conclusions. When we have 5 or 6 more results across a few games I think we can start to look at patterns.

 

[Dr_Fartypants]

It SEEMS from your descriptions that you are trying to ignore this type of stutter but instead are looking for general variations from the total average frame rate. To me this means that you are looking normal drops in frame rate from the average which you would get with any GPU. If this is the case then I would be the first to say that the measurement isn't worthless but it doesn't imo measure microstuttering?

'Microstutter' takes place at very short timescales.

To make it clear: to make the index I don't measure variation away from the TOTAL average framerate (which would be just measuring framerate consistency as you say...), I measure variation away from the LOCAL averaged framerate.

What I do, for each frame, is look at the frametime for the surrounding 9 frames (that frame, 4 before it, and 4 afterwards). From this I determine the LOCAL averaged framerate. This smooths out local solution noise, which is that we are trying to measure. I then compare the frametime of that particular frame against the 'averaged' frametime. This is a typical method for determining local data noise (eg turbulence intensity in fluid dynamics).

Note that this 'averaged' framerate will still vary (a lot) during the course of the benchmark. All framerate indicators that you see in games (like FRAPS or in-game indicators) will use some kind of local time averaging, otherise they would constantly be changing value every 1/50th of a second or so.


As for culling the worst 2% of values - we're looking for purely local effects (microstutter) rather than global hitching (paging / regular stutter etc). By removing the worst 2% of values we eliminate most of the jumps caused by paging or other global phenomena.

For more information on the procedure I use, take a look at the readme

Microstuttering from my own experience and understanding happens when the second GPU fails to "hit" the next frame in time and so the output from the second GPU is ignored. This is effectively (imo) just like the problem with VSYNC when the frame rate drops below the monitor SYNC rate, the frame is ignored and you get a stutter.

That's one aspect of it, but 'microstutter' in general is just 'framerate output irregularity'. Of course, if one frame output is totally missed, that will appear as a significant irregularity. Since this would happen frequently and on a short timescale, it would be picked up by my program, and would add to the microstutter index.

 

[DragonRR]

What I do, for each frame, is look at the frametime for the surrounding 9 frames (that frame, 4 before it, and 4 afterwards). From this I determine the LOCAL averaged framerate. This smooths out local solution noise, which is that we are trying to measure. I then compare the frametime of that particular frame against the 'averaged' frametime. This is a typical method for determining local data noise (eg turbulence intensity in fluid dynamics).

OK. although I'm sure you JUST added your step by step bit

I stand (or sit in this case) corrected

 

[lay-z-boy]

Tests on an xps m1730 with 4gb, stock clocks on 8800m gtx sli + single, 177.39 drivers. Using a t9300 for now, might try the x9000 later.

Remember, this system uses a PM965 nothbridge (single 16x) with a nForce 100 chip splitting that to two 16x.

cod4 chinatown @ 1920x1200, everything maxed out with 2x aa

Single



SLI



hl2 custom map with high res textures + hdr

1920x1200, everything maxed, 16x qaa

Single



SLI



Will add a few more games and try another driver later.

 

[Scougar]

REF: Lay-z-boy "cod4 chinatown @ 1920x1200, everything maxed out with 2x aa SLI with 32 percent stutter"

How does it actually play though? Does it seem to "stutter" in the referenced setup?

Matthew

 

[Dr_Fartypants]

Fantastic Thanks for the results, lay-z-boy

It certainly looks like there is some form of microstutter going on there which is due to the SLI configuration. Although, 20% in HL2 isn't too bad, really. Certainly the increase in raw framerate will more than make up for any decrease in apparent smoothness due to the microstutter. So, in this case, SLI seems a worthwhile bet.

 

[Dr_Fartypants]

How does it actually play though? Does it seem to "stutter" in the referenced setup?

At 112fps average I would doubt it. Even if the microstutter was far worse (closer to 100%) it would still 'appear' at the very worst to be half this (about 56fps average). This is still fairly smooth.

The strong scaling of performance with SLI should have overcome the effect of microstuttering. But of course it's subjective, so maybe lay-z-boy has a different opinion.

Remember, 'microstuttering' isn't "stutter" per se, like paging. It's just an apparent reduction in framerate due to uneven output of frames.

 

[lay-z-boy]

REF: Lay-z-boy "cod4 chinatown @ 1920x1200, everything maxed out with 2x aa SLI with 32 percent stutter"

How does it actually play though? Does it seem to "stutter" in the referenced setup?

Matthew

In sli configurations I experienced a small amount of stutter even though the fps was higher. This is down to the drivers though, I've just changed to the 177.66's and the stutter that was present has now gone.

I'm going to do the benchmarks again because of this and update my original post accordingly.

 

[Dr_Fartypants]

I'm going to do the benchmarks again because of this and update my original post accordingly.

Sounds good. Could you leave the originals there though, so we can see what (if any) difference the new drivers make.

It's nice to see how much is down to software interfacing and how much is down to the hardware itself.

 

[lay-z-boy]

Sounds good. Could you leave the originals there though, so we can see what (if any) difference the new drivers make.

It's nice to see how much is down to software interfacing and how much is down to the hardware itself.

Fine, I'll leave the first post there.

Same settings as before but now on the 177.66 drivers.

COD4 Single



COD4 SLI



HL2 Single



HL2 SLI



Audiosurf settings: 4x aa, 16x af, 1920x1200, premium details
Smash brothers brawl - melee theme remix

Audiosurf Single



Audiosurf SLI



UT3 settings: 2x aa, 192x1200, ingame settings maxed out, hardware ppu physx enabled
VCTF Corruption

UT3 Single



UT3 SLI



Methinks something went wrong with the sli hl2 test but I've run it twice and its showed the same results. :/

 

[Dr_Fartypants]

Looks like drivers can have a big impact then. This suggests that the problem is at least 'solvable'. Which is good news...



On a separate note, I found this graph, posted by Sampsa over at XS, to be quite interesting also:

 

[marscay]

as i said earlier in the thread Sampsa has some interesting news/results regarding microstuttering and the ATI 48xx series.

those cheap GX2's don't seem as such a great deal anymore

 

[PCZ]

We have been living with this problem for a long time.
Not talked about much but very real all the same.

That it seems to be cured with the newer ATI cards is excellent.

 

[Bobly]

No new results?

 

[MR.B]

I had micro stutter on my 9800GX2, but it was purely to do with the silly 512MB usable VRAM, and my screens 2560x1600 res. And i think this is the main problem for this issue, not enough memory. Which will effect non-SLI/crossfire setups exactly the same.

Didn't get it with the 8800GTX, or my GTX280 which both have more usable VRAM. When i get a 4870 X2 2GB i do not expect to have micro stutter issues.

My thread here is a little related to this issue, and the micro stutter/lower frame rates on my previous GX2 compared to my GTX280 are purely related to not enough VRAM on the GX2 when using high resolutions.
It runs out of memory and performance is crippled as it starts switching to system RAM, which apart from causing ridiculously low frame rates it also has the effect of micro stutter. When you begin reaching the 512MB usable VRAM limit, when the frame rates are not yet affected, at this point you begin to get micro stutter - then turn on some AA and you run out of memory and it's slideshow time.
And i'll update that thread when i get the X2.

 

[Dr_Fartypants]

I had micro stutter on my 9800GX2, but it was purely to do with the silly 512MB usable VRAM, and my screens 2560x1600 res. And i think this is the main problem for this issue, not enough memory. Which will effect non-SLI/crossfire setups exactly the same.

"microstutter" is a completely different phenomenon to the VRAM to system-RAM paging that you describe. The former is due to uneven frame output, and the latter due to not having enough video memory in which to fit all the textures.

Any time you get paging though it really does kill performance. I can well imagine that the GTX280 is a lot better than the GX2 in some games, on a 30" screen - particularly with AA enabled.