I've been running 3Dmark01, 03, 05, and PCmark05 lately and there are a couple of things that make me think that it is crap.

FOR A BENCHMARK ON WINDOWS SYSTEMS ONLY:
If the benchmark is designed to test graphics; you SHOULD:
- be able to choose the rendering engine/language be it with OpenGL or DirectX. Not just one. Most cards sold nowadays support at least OpenGL 1.5; therefore, there's absolutely no reason why you can't pick besides whether it's hardward or software driven.

- it should not get "confused" when it detects something newer and doesn't know what to do with it. I don't have what the PS 3.0, 2.0, 1.1 and 1.0 specifcation white papers; but I would guess that a newer version would be adding features that earlier version don't have thus making it backwards compatible. Therefore; how can it get "confused" when it doesn't recognize it or when it detects more? And then show that it is AT least at the maximum level that the program requires/detects and not run?

- It should be able to record the number of dropped frames due to either the video card not being able to render it (lack of feature support) OR because it decided that it takes too long/timed out/too intesive for the card. (yes, ATi and nVidia cards will start dropping frames if it decides that it is taking too long to process resulting in higher somewhat-artifical frame rates.)

I'm no expert on programming, but I would think that if a benchmark is sending data to be rendered by the video card, with n number of frames, it should expect n number of frames back; and that if n - n does not equal =, introduce a new variable, m, such that m is the number of dropped frames and that n - (n+m) = 0.

FOR BENCHMARKING SOFTWARE IN GENERAL
- should have support for all available OS/CPU combinations. The only one that I've seen do it really well is distributed.net.

Now I realize that 3Dmark 05 and 03 explicitly say that they are "gaming" benchmarks, but that shouldn't be a reason for it a) not to be running on other systems, and b) that the rules of benchmarking applies regardless.

Quote: Originally Posted by alpha754293 I've been running 3Dmark01, 03, 05, and PCmark05 lately and there are a couple of things that make me think that it is crap. FOR A BENCHMARK ON WINDOWS SYSTEMS ONLY: If the benchmark is designed to test graphics; you SHOULD: - be able to choose the rendering engine/language be it with OpenGL or DirectX. Not just one. Most cards sold nowadays support at least OpenGL 1.5; therefore, there's absolutely no reason why you can't pick besides whether it's hardward or software driven. - it should not get "confused" when it detects something newer and doesn't know what to do with it. I don't have what the PS 3.0, 2.0, 1.1 and 1.0 specifcation white papers; but I would guess that a newer version would be adding features that earlier version don't have thus making it backwards compatible. Therefore; how can it get "confused" when it doesn't recognize it or when it detects more? And then show that it is AT least at the maximum level that the program requires/detects and not run? - It should be able to record the number of dropped frames due to either the video card not being able to render it (lack of feature support) OR because it decided that it takes too long/timed out/too intesive for the card. (yes, ATi and nVidia cards will start dropping frames if it decides that it is taking too long to process resulting in higher somewhat-artifical frame rates.) I'm no expert on programming, but I would think that if a benchmark is sending data to be rendered by the video card, with n number of frames, it should expect n number of frames back; and that if n - n does not equal =, introduce a new variable, m, such that m is the number of dropped frames and that n - (n+m) = 0. FOR BENCHMARKING SOFTWARE IN GENERAL - should have support for all available OS/CPU combinations. The only one that I've seen do it really well is distributed.net. Now I realize that 3Dmark 05 and 03 explicitly say that they are "gaming" benchmarks, but that shouldn't be a reason for it a) not to be running on other systems, and b) that the rules of benchmarking applies regardless.

One major problem with measuring dropped frames and elements is that the drop occurs within the graphics card and the only current point where such data could be gathered consistently would be on the graphics card prior to output to the monitor although I'm certain somebody could effectively write a query for the graphics card that would not severely impact performance at the time of the query.

I have to slightly correct you on the remark regarding ATI and nVidia cards; they are not dropped purely due to processing duration. Such frames are dropped due to an inability to complete the frame based upon a) whether the frame following it has been completed in time for display prior to the current frame (and whatever mechanism that there is to internally determine if the dropped frame will be readily noticable on-screen), b)if the elements contained within the frame surpass a threshold beyond which the hardware behavior becomes conducive to instability.

My two cents (what a graphics benchmark should specifically be able to measure):

FPS
Resolution
AntiAliasing settings
Dropped Frame
Dropped Elements
Dropped Textures
Render elements omitted (this frequently happens with many popular cards due to polygon intersection that occurs outside of the field of view)

Is this like the 3rd or 4th thread that you started regarding benchmarking???

Quote: Originally Posted by Highperf15 Is this like the 3rd or 4th thread that you started regarding benchmarking???

It looks like it's his first in the Benchmarking >Apps forum

All I can say on this subject is you need to match the benchmark to the intended use/application of the Computer.

Here is something that may a bit more in line with your rigs:

http://www.anandtech.com/IT/showdoc.aspx?i=2291&p=20

Anandtech has plenty of other articles on these types of benches you may find more "Usefull"
http://www.anandtech.com/IT/showdoc.aspx?i=2291&p=20

I personally would never dream of running something like MYSQL on this rig!

Here goes again...look man, nobody really cares what you think benchmarking software should do.

I disagree with you on your last comment.

Quote: Originally Posted by alpha754293 that shouldn't be a reason for it a) not to be running on other systems, and b) that the rules of benchmarking applies regardless.

Let me also quote the Futuremark website.

Quote: By combining high quality 3D tests, CPU tests, feature tests, image quality tools, and much more, 3DMark05 is a premium benchmark for evaluating the latest generation of gaming hardware.

Evaluation of the lastest gaming hardware....mmmm, interesting. The latest generation of gaming hardware is best ultilised on Windows based systems.

Windows is a great OS for gaming, soon to get even better. I don't see any games for Solaris using capabilities of the latest gaming hardware.

Do you think thats why the software isnt so universal in its choice of OS? 99.9% of people wouldnt associate Solaris or Linux or Unix based systems with "the latest gaming hardware".

Also, what do you hope to achieve through this thread? Intelligent discussion? Respect? My guess is you were just starved of attention since BA closed your first thread.

Quote: Originally Posted by $SOLID$ Necro All I can say on this subject is you need to match the benchmark to the intended use/application of the Computer.

Thus the existance of differentiated benchmarks...I agree in that regard.

Quote: Originally Posted by $SOLID$ Necro I personally would never dream of running something like MYSQL on this rig!

Nor would I on mine.

The characteristics stated at the start of this thread drive toward a unified/commonized benchmarking mechanism from which a "gaming index," a "business index," etc. could be derived based upon selecting particular numbers and weighing those together while ignoring others. However, there are a few certain characteristics that no publically available benchmarks of which I know are currently capable of reporting. I'm certain that it's the lattermost of this to which Alpha is leaning.