The Benchmark King: How Futuremark Keeps Performance Score

[~Mike~]

The Benchmark King: How Futuremark Keeps Performance Score

When it comes to benchmarking, Oliver Baltuch concedes, rather mildly, there can be “a lot of religious arguments”. That’s for sure: benchmark scores have long been powerful marketing tools and have thus been prone to accusations of skulduggery and shenanigans by vendors scorned by low marks. But, as president of the highly respected benchmarking group Futuremark, Baltuch is a proponent of fair play and when I met the Canadian in London recently, he talked frankly about the issues surrounding these tests and the importance of transparency

“When you look at what goes in to a computer and how the processor is built, the computer user doesn’t care,” he says. “They’re interested in what they can do with it. It’s a tool. Whether it’s AMD, nVidia or ARM shouldn’t matter that much, but what matters is [the ability to fulfil tasks]. What a benchmark is for is to take a machine and place it into human terms — because we’re human, not machines. A benchmark will allow you to compare two machines that are almost identical, to see the differences.”

Read the full article here
http://www.idgconnect.com/abstract/8474/the-benchmark-king-how-futuremark-keeps-performance-score

 

[Disco_P]

I thought Matt was the Benchmark king?

 

[~Mike~]

I thought Matt was the Benchmark king?

He does love his benchmarks.

 

[LtMatt]

Futuremark do tend to make the best and fairest benchmarks to be fair. It's normally pretty even and no shenanigans like heavily tessellating things that can't be seen on screen.

 

[pgi947]

Futuremark do tend to make the best and fairest benchmarks to be fair. It's normally pretty even and no shenanigans like heavily tessellating things that can't be seen on screen.

Lol you must be kidding your self if you think 3dmark (any of them) can't be manipulated to swing heavily in one vendors favour.

 

[LtMatt]

Lol you must be kidding your self if you think 3dmark (any of them) can't be manipulated to swing heavily in one vendors favour.

I'm sure it can be but I've not noticed it on any of the recent ones and i know that if i attempt to alter any settings i shouldn't in CCC it picks up on that when i validate and will say xxx modified etc. Can't speak for Nvidia and if it detects those things, but it certainly does for AMD guys.

 

[pgi947]

It depends how you implement them, iirc using Tess mods on amd cards for 3dmark was easy to do and easy to pass validation. Things like altering IQ are rife on both sides of the fence, the winner is always the guy that knows how to fine tune all of these 'tweaks' the best.

In short I wouldn't put futuremark stuff as the be all and end all of accurate benching, its something that doesn't exist outside of a highly conttolled environment.

 

[LtMatt]

It depends how you implement them, iirc using Tess mods on amd cards for 3dmark was easy to do and easy to pass validation. Things like altering IQ are rife on both sides of the fence, the winner is always the guy that knows how to fine tune all of these 'tweaks' the best.

In short I wouldn't put futuremark stuff as the be all and end all of accurate benching, its something that doesn't exist outside of a highly conttolled environment.

Well it's definitely the best of a bad bunch then as it's the only one that can detect CCC tweaks or certain attempts to cheat.

As for Tess Mods first I've heard, but i don't really pay much attention to cheating and all the complex methods people use to get a few notches higher on a leaderboard.

I just use diagnostic mode in Win8.1 and put everything up to 100%/highest and away i go. #Technical

 

[Rangerjr1]

I thought Matt was the Benchmark king?

lol

 

[Rangerjr1]

It depends how you implement them, iirc using Tess mods on amd cards for 3dmark was easy to do and easy to pass validation. Things like altering IQ are rife on both sides of the fence, the winner is always the guy that knows how to fine tune all of these 'tweaks' the best.

In short I wouldn't put futuremark stuff as the be all and end all of accurate benching, its something that doesn't exist outside of a highly conttolled environment.

You wont get away with tess tweaks unless there's something you can do about that i havent heard of. Nvidia on the other hand gets away with LOD tweaks which is kind of stupid, and oh my nv owners abuse it.

 

[mmj_uk]

Futuremark do tend to make the best and fairest benchmarks to be fair. It's normally pretty even and no shenanigans like heavily tessellating things that can't be seen on screen.

Can't resist a cheap dig eh?

If anything Futuremark (emphasis on Future) should have been using lots of heavy tessellation to simulate future game engines, why hold the levels back just because AMD can't keep up?

They're only fair in the sense that they've chosen preset levels that work equally well on both brands of cards and ignoring advantages such as NVidia's tessellation, if they'd have gone with lots of heavy tessellation they would have had half of their userbase (AMD fanboys) slandering them and it would have been bad for business.

 

[LtMatt]

Is it just me or does every single post from mmj_uk look exactly the same as the previous one? Every post includes the word 'AMD fanboys'.

 

[Rangerjr1]

Can't resist a cheap dig eh?

If anything Futuremark (emphasis on Future) should have been using lots of heavy tessellation to simulate future game engines, why hold the levels back just because AMD can't keep up?

They're only fair in the sense that they've chosen preset levels that work equally well on both brands of cards and ignoring advantages such as NVidia's tessellation, if they'd have gone with lots of heavy tessellation they would have had half of their userbase (AMD fanboys) slandering them and it would have been bad for business.

But they'll allow LOD changes on nv, but the second you disable tess futuremark goes all INVALID.

 

[David Bisset]

Can't resist a cheap dig eh?

If anything Futuremark (emphasis on Future) should have been using lots of heavy tessellation to simulate future game engines, why hold the levels back just because AMD can't keep up?

They're only fair in the sense that they've chosen preset levels that work equally well on both brands of cards and ignoring advantages such as NVidia's tessellation, if they'd have gone with lots of heavy tessellation they would have had half of their userbase (AMD fanboys) slandering them and it would have been bad for business.

I've no way of knowing that future game engines will increase tessellation use heavily over current engines, do you? As with all benchmarking, the best test is always testing real world uses and if Futuremark can emulate a range of these then fair play to them - still not as good as seeing how actual games run, but then as always what game is chosen becomes very significant. Personally I enjoy playing with benchmarks on both my computers, but when looking to buy new kit I look at benchmarks for the games I actually play.

Guessing what is going to come next is not benchmarking, it's just having fun with made up stuff. And heavily tessellating water to the point it's a bottleneck for both makers cards is just being an idiot.

Edit: Given I mostly buy lower end/older kit I'm not really as excited by all the battles over who has the fastest benchmark score though, so maybe on this forum my view isn't particularly relevant.

 

[triss]

I've no way of knowing that future game engines will increase tessellation use heavily over current engines, do you? As with all benchmarking, the best test is always testing real world uses and if Futuremark can emulate a range of these then fair play to them - still not as good as seeing how actual games run, but then as always what game is chosen becomes very significant. Personally I enjoy playing with benchmarks on both my computers, but when looking to buy new kit I look at benchmarks for the games I actually play.

Guessing what is going to come next is not benchmarking, it's just having fun with made up stuff. And heavily tessellating water to the point it's a bottleneck for both makers cards is just being an idiot.

Edit: Given I mostly buy lower end/older kit I'm not really as excited by all the battles over who has the fastest benchmark score though, so maybe on this forum my view isn't particularly relevant.

Your view as relevant as anyones imo,, you are still a user of said hardware

 

[LtMatt]

^ Agreed.

 

[scotsdragon]

Why does it always have to degrade into fanboys this and fanboys that, im sure most are like my self and just want value you for money.

As you guys can see from my sig I have an NVidia card just now but I have had many cards on both sides and since im looking at changing again im measuring up both potential cards from either side.

And I must say they AMD cards look pretty damn good... at maybe 5fps on Average loss(in some games and even in sometimes better in others), Compared to the extra 200 pounds it would cost me to go Nvidia .......... mmm 200 pounds in my pocket or 5fps mmmmmm (unless a missive price drop I know where my cash would go).

Since I have till the start of September to make up my mind like most I will keep comparing


Benchmarks are good in some cases but completely pointless in others

do I sit watching a FPS counter while gaming No! I play the game, I only look at FPS if the game starts playing crap at the settings I like.

 

[whyscotty]

do I sit watching a FPS counter while gaming No! I play the game, I only look at FPS if the game starts playing crap at the settings I like.

^^^
This

 

[LtMatt]

Yes.

 

[Uncle Petey]

I'm surprised Matt can even see the game behind all those graphs