Microstutter - fact or fiction? Let's find out once and for all...

Dr_Fartypants · 13 Jul 2008 at 14:48

There is much debate about 'microstutter' (which should really be called 'uneven frame output' but anyway) when using multi-GPU setups. Many people report that they notice this when using their multi-GPU systems. A few have produced results 'proving' its existence, whereas others swear down that they have never experienced it, and/or question the validity or applicability of the very few numerical results actually available.

Anyway, to help answer the question, I have written a very simple little program which analyses FRAPS benchmark outputs to determine the degree of microstutter present. You can download the program here:

http://www.mediafire.com/download.php?2moyugibhez (thanks lightnix...)

Of course there will be some non-uniformity of frame output, even in single GPU setups. Also it is clear that the degree of non-uniformity changes from game to game. So, what would really be interesting is to find out by how much using a multi-GPU setup increases the level of microstutter, if at all.

Instructions for use are included in the readme.txt file (pasted below).

To get started, here are a couple of results I gathered to check that the thing was working. These were both gathered stood still, looking out across a vista (there was some dynamic stuff happening in the distance in each case however):

System:
e6600 @ 3.4Ghz
GTX280 @ 650/1107

HL2 ep2:

Crysis (1920*1200 high)

Anyway, single GPU results would be interesting, but the multi-GPU results should be the real eye-openers, one way or another. It would also be nice to see if there is a difference between SLI and xfire as far as microstutter goes. If so, that would be a real selling point in my eyes.

readme.txt included in file said:
This little program attempts to quantify the degree of "microstutter" in a particular FRAPS benchmark output. Microstutter has become the popular term for describing uneven frame output, often present in multi-GPU setups when running with an alternating frame rendering (AFR) configuration. Its effect, when present, is to make a given framerate appear 'less smooth' than an equivalent framerate with a single GPU setup.

*** How to use the program ***

1) Make a folder wherever you like, and copy across the microstutter_test.exe file

2) Download and install FRAPS (http://www.fraps.com/). In FRAPS, go to the 'FPS' tab, and select the 'Frametimes' tick-box (located at the bottom left)

3) Enter your game with FRAPS running, and at a suitable point, run a FRAPS benchmark (default key is F11, I think...). Run the benchmark for a suitable length of time (say 30 - 60 seconds) during regular gameplay (no loading screens)

4) Go to the FRAPS/benchmarks folder, and find the appropriate *.csv file for the benchmark you just ran.

5) Copy this *.csv file to the microstutter test folder and rename the *.csv file to: input.csv

6) Run the microstutter_test.exe program, and observe the microstutter index. The higher the index, the greater the effect of microstutter during the benchmark.

*** How NOT to use the program ***

1) No not use with (double buffer) vsync enabled.
The microstutter index will always appear to be zero, since double buffer vsync switches discretely between constant framerates. However, if you have triple buffer vsync working properly, and your framerate stays below your monitor refresh rate (usually 60), then it might be interesting to see by how much the microstutter is reduced.

2) Do not use in games with a maximum framerate cap (unless your framerate stays below this)
As above, a fixed maximum framerate will lead to a zero microstutter index. Microstutter is not an issue anyway if you are operating at your maximum monitor refresh rate, or at a fixed framerate.

3) Do not run a benchmark while the game is loading, particularly if a separate loading screen is shown. In-game loading and paging should be accounted for to some extent (see below) but it is best to avoid this if possible. A good microstutter index can be obtained simply staring into space and doing nothing.

*** How does the program work? What is the microstutter index, exactly? ***

In short, the degree of microstutter is calculated for every frame by comparing its variation away from the local average frametime. This value is scaled by the local average frametime to obtain a non-dimensional index. The reported 'microstutter index' is the average of these values over the benchmark, multiplied by 100.

So, the smaller the microstutter index the closer each frame is to the 'smoothed' average framerate, and the less 'noisy' the data.

You can think of the microstutter index as being the (average) percentage varation of each frame away from the stabilised local framerate.

...Step by step:

The local average frametime is obtained by taking the average of the frametime for 4 frames in either direction (for a total of 9 contributions to the average). This is not calculated for the first 5 or last 4 frames, and so these frames are not included in the microstutter statistics.

The variation away from the average frametime is calculated for each valid frame. To help avoid the final data being skewed by large 'spikes' due to paging etc and other game related events, the largest 2% of local varations are culled, and no longer appear in statistics.

The local variation in frametime is divided by the local average frametime to produce a small nondimensional number (local microstutter index).

The average, over valid and non-culled frames, is taken for the local microstutter index. This number is multiplied by 100 to produce a more 'user-friendly' range of numbers.

*** Anything else? ***

Well, don't abuse me for the noobishness of the program. I'm not a computer scientist or software engineer or anything like that.

If you find any real bugs, let me know on the ocuk forums thread (see http://forums.overclockers.co.uk/ and look for Dr_Fartypants)

If you want to create a nice visual interface for the program, or otherwise improve it, then go right ahead! You can use the executable, or I can pass you the (horrible fortran90) source if you like. Just let me know what you're doing is my only requirement.

If you want to find me then I can usually be found at OcUK forums (Dr_fartypants), or hardforums (arseface).

rounddodger · 13 Jul 2008 at 15:17

GRID 1920 x 1080 Ultra settings 4xMSAA

Score seems a bit crappy to me? I tried this with VSYNC on as well for comparison and it was higher (ie. worse?) even though 60fps VSYNC is smoothest by a country mile! Silky silky smooth in fact.

Dr_Fartypants · 13 Jul 2008 at 15:22

The amount of stutter varies game-by-game anyway, so I wouldn't worry about that. The key thing will be to see if multi GPU setups also have a similar level of microstutter (<10% say), or if they're much much higher (ie 30%+).

But it's certainly odd that you get a larger value with vsync on... I'll have a look how FRAPS handles it, whether it records the time at which the frame is displayed or is sent to the buffer. The other possibility is that the monitor is not dead-on 1/60 th second update every time.

edit - yeah, that is strange, with vsync. I just tested HL2 with vsync enabled, and I got a microstutter index of 0.56. I guess it could be down to the monitor. Maybe a TV (even a good one like yours) isn't as precise in its update times, since it doesn't need to be?

rounddodger · 13 Jul 2008 at 15:30

Dr_Fartypants said:
The amount of stutter varies game-by-game anyway, so I wouldn't worry about that. The key thing will be to see if multi GPU setups also have a similar level of microstutter (<10% say), or if they're much much higher (ie 30%+).

But it's certainly odd that you get a larger value with vsync on... I'll have a look how FRAPS handles it, whether it records the time at which the frame is displayed or is sent to the buffer. The other possibility is that the monitor is not dead-on 1/60 th second update every time.

Here is the results for reference with VSYNC in game enabled, not sure what type of vsync is enabled, double, triple whatever!

Dr_Fartypants · 13 Jul 2008 at 15:34

rounddodger said:
http://www.rounddodger.plus.com/files/grid_vsync.gif

Here is the results for reference with VSYNC in game enabled, not sure what type of vsync is enabled, double, triple whatever!

Yep - very strange. Well, what can I say? It could be due to the TV you're using, but that's a guess. I just tested HL2 and crysis with vsync and both come out with values less than 1%.

Still, for a really noticable microstutter I would be expecting a value of around 30%, or even higher.

rounddodger · 13 Jul 2008 at 15:55

Dr_Fartypants said:
Yep - very strange. Well, what can I say? It could be due to the TV you're using, but that's a guess. I just tested HL2 and crysis with vsync and both come out with values less than 1%.

Still, for a really noticable microstutter I would be expecting a value of around 30%, or even higher.

Ahh may just be GRID, just ran HL2EP1 and 1080p MAX detail VSYNC 4xAA 16xAF I get :-

Average Microstutter index :- 0.251

Try running GRID and see what you get

Dr_Fartypants · 13 Jul 2008 at 16:10

rounddodger said:
Try running GRID and see what you get

I don't have grid, I'm afraid. But yeah, sounds like it's just an issue with GRID.

Surprising though, as I would have thought vsync would be so closely tied to the monitor refresh rate as to be virtually the same in all games.

edit - if you want, you can send me the file and I'll have a look at it more closely. arseface1 (at) hotmail (dot) com

Talrinys · 13 Jul 2008 at 16:13

Fantastic program and thread, this is gonna be hugely interesting when the 4870x2 comes out!

Big.Wayne · 13 Jul 2008 at 16:18

Looks interesting Dr_Fartypants!

rounddodger · 13 Jul 2008 at 16:19

Dr_Fartypants said:
I don't have grid, I'm afraid. But yeah, sounds like it's just an issue with GRID.

Surprising though, as I would have thought vsync would be so closely tied to the monitor refresh rate as to be virtually the same in all games.

edit - if you want, you can send me the file and I'll have a look at it more closely. arseface1 (at) hotmail (dot) com

Sent!

lay-z-boy · 13 Jul 2008 at 16:25

Nice program. Will get round to a few tests of 8800m gtx sli vs single on the laptop and 8800gtx sli vs single on the desktop later on.

Should be interesting.

Dr_Fartypants · 13 Jul 2008 at 16:32

rounddodger said:
Sent!

Just had a look, and wow - it really is all over the place isn't it? Really not sure what's going on with GRID then. I suppose it could be an issue with the FRAPS readout or something.

lay-z-boy said:
Nice program. Will get round to a few tests of 8800m gtx sli vs single on the laptop and 8800gtx sli vs single on the desktop later on.

Should be interesting.

That would be great

Single vs SLI (and x-fire) comparisons are what we really need to see to settle the debate

Sniper123 · 13 Jul 2008 at 18:14

lay-z-boy said:
Nice program. Will get round to a few tests of 8800m gtx sli vs single on the laptop and 8800gtx sli vs single on the desktop later on.

Should be interesting.

Cool!

I'm to lazy

Devil_Spawn · 13 Jul 2008 at 18:37

we need results from both single gpu and multi gpu, perhaps even the same system running with and without xfire/sli.

Dr_Fartypants · 13 Jul 2008 at 18:44

Devil_Spawn said:
we need results from both single gpu and multi gpu, perhaps even the same system running with and without xfire/sli.

Yep - that would be ideal.

If I had a multi-GPU system, I would run them myself. I don't though, so that was one of the main reasons for making a little app that other people could run

slimsymeat · 13 Jul 2008 at 21:41

I'm depressed now.. Ran it with rFactor.. About 10,000 frames with and avg microstutter index of 45.650. That's bad. Normally I cap the fps and vsync at 60hz so it runs smooth, I guess that's the reason why eh!! :mad:

Devious · 13 Jul 2008 at 22:01

I ran 6 games single card and crossfired. Ran each test twice and results were consistant.

Conan

7.972 / 8.017

Crysis

25.805 / 21.954

Devil may cry bench

2.222 / 39.859

HL2 mod FrontlineForce

6.445 / 17.051

GRID

4.195 / 8.632

Mass Effect

4.576 / 3.258

Seems i picked some bad games to test as they dont appear to benefit from crossfire and thus the index was similar ...

ubern00b · 13 Jul 2008 at 22:04

http://www.ubern00b.net/microstutter.rar

Hosted for ya if you want to update the main post.

dbmzk1 · 13 Jul 2008 at 22:08

You don't need a program to test for it you can SEE it!

Bobly · 13 Jul 2008 at 22:09

/me bookmarks this thread.

Great tool

I wish I could help but my out-dated laptop isn't gonna contribute much