BigAkita has a nice little article up outlining his opinion on a recent article by K. Bennett recently published in CPU Magazine. My opinions are right alongside Brett's on this topic.


Check it out here.

Quote: 7800GT video card relates to the "experience" an end user on a Socket A motherboard, 512K of value RAM, Barton processor, and Ti4800 video card.

Isnt that A misprint? Doesnt BA mean 512MB?

Yep, got to fix that, thanks. Now you see what I mean when I say we aren't perfect.

Quote: The idea of scraping, or at least back-seating

Who's yer editor?

HAHAH my fault... fixing that now. Spellcheck can't catch them all I guess.

Well, the only thing I can say in regards to Kyle Bennet's idea is that it is just plain silly. Not to mention quite pompous.

In a means of comparison let's take the automobile industry. Suppose that instead of using radar, crash test dummies, computers, and all the other testing equipment to check safety, speed, equipment integrity we just decided to use the best judgement of the people testing the cars. Because afterall they have been doing it for years. Haven't they?

Surely the result would be not only a drop in auto sales, but more accidents. There would be a lot of cars that are not only unsafe, but faulty on our highways and biways.

Using benchmarks to test computers does gives a sense of standardized tests. Until there is one specific way to test all computers in the same fashion, it is the closest best way to achieve that in the meantime.

Quote: Originally Posted by Foe Well, the only thing I can say in regards to Kyle Bennet's idea is that it is just plain silly. Not to mention quite pompous. In a means of comparison let's take the automobile industry. Suppose that instead of using radar, crash test dummies, computers, and all the other testing equipment to check safety, speed, equipment integrity we just decided to use the best judgement of the people testing the cars. Because afterall they have been doing it for years. Haven't they? Surely the result would be not only a drop in auto sales, but more accidents. There would be a lot of cars that are not only unsafe, but faulty on our highways and biways. Using benchmarks to test computers does gives a sense of standardized tests. Until there is one specific way to test all computers in the same fashion, it is the closest best way to achieve that in the meantime.

Damn it Foe.........You beat me to using the "auto company" example . I've been reading Kyle's evals today there's enough bias-fluff to fill more than a few pillows.........BA hit it on the mark in regards to testing standards

Yup... gotta agree with Foe here. I read Kyles article and while it was moderately entertaining, it is ridiculous.

You have to use some sort of benchmarking software to determine baselines, as well as thresholds of equipment, old and new.

I guess you could go with a "feeling", but it doesn't stack up very well in our IT profession.

Nice try, Kyle.

I thought this article appeared more like a personal attack rather than a different state of opinion. If you read HardOCP's latest reviews, they are still using benchmarks which hardcore gamers look for. I think, and I could be wrong, but that his point was that benchmarks are not the sole factor that decides how good a system is. There are other factors such as setup speed, ease of use, clear and concise instructions, user experience etc that determine how good a system is. I don't subscribe to CPU Magazine so I could be off here.

Comments like "There are actually a couple of sites that agree with him, but they are no more than sidekicks begging for approval and have their heads so far up Kyle's posterior that they can tell what he had for lunch." show unprofessional journalism IMO. HardOCP has their readers, and PimpRIG has their own. Readers will always gravitate towards a site which they can relate with. There is no so called "standard" that determines how others must run their site.

I think people would have respected your article more if you explained why you felt Kyle's view is wrong and proposed a better view instead. As is stands, this article sounds like it was written by someone that had a bad night of constipation.

Ian Bell
Designtechnica.com

Ratings and benchmarks are not a 100% accurate and universal way of gauging and comparing performance.

To me, this would suggest that more standardized, objective methods of testing should be developed. Then, the objective scores can be presented along with a reviewer's opinion -- because a reviewer's opinion is valuable too, but only to the degree that his/her interests and standards match mine.

To suggest that objective measurements should be thrown out simply because they're not 100% accurate and universally comparable is an, um, interesting approach. To my way of thinking, relying entirely on a single reviewer's subjective opinion would only serve to amplify the flaws he seeks to correct.

As for BA's approach: According to his byline in CPU, Kyle is known for his "strong opinions and stating them in a no-nonsense manner." So if such a person does come up with nonsense, he should surely be able to handle -- even applaud -- a dissenter's strong opinion stated in a no-nonsense manner.

I thought BAs' article was great, hit the nail right on the head. Kyles needs a reality check.

I occasionally surf over there, but the forums aint got 'nutin on PR

Quote: Originally Posted by Ioman II think people would have respected your article more if you explained why you felt Kyle's view is wrong and proposed a better view instead. As is stands, this article sounds like it was written by someone that had a bad night of constipation. Ian Bell Designtechnica.com

I appreciate your comments. Just like Kyle said wait for part two, I am doing the same.

Kyle's been touting the "experiece" review for a loong time now. I would have to say though, and I'm sure BA would agree, is that while the reviewer's experiences are important to a well-rounded review, they shouldn't be the baseline. There are too many factors to an "experience".

For instance:

You could get a mediocre-performing part, but because your wife's been on your ballz to get the grass cut, this mediocre-performing part gets a bum rap cuz it's taking too long to complete the review. Unfair for the sponsor, manufacturer, and consumer alike. The reviewers bad mood would probably translate into a bitter opinion of the part. It's human nature. (Why do you kick the cat after getting in a fight with your significant other??)

Benchmarks, while flawed, at least give you raw numbers to compare. besides...how is your experience of Unreal2004 at 120FPS going to comapre with Unreal2004 at 170FPS? Which card performs better?

Duh...looks OK to me...

Well I have gotten a couple of issues of Computer Power User. When I saw this article I couldnt tell you how utterly pissed I was. He is crying over something that has been going on for years. If he feels so strongly about this why doesnt his site follow it? Now im in college and the food isn't that great and lets just say page 30 felt better on my a$$ than the tp they offer here in the bathrooms.

Kyle Bennett is editor-in-chief of HardOCP.com, one of the largest and most outspoken PC-enthusiast sites on the Web. HardOCP.com is geared toward users with a passion for PCs and those who want to get cutting-edge performance from their systems. Beware, though, Kyle is known for his strong opinions and stating them in a no-nonsense manner while delivering some of the most in-depth reviews and PC hardware news on the 'Net.
---------------------------------------------------------------------------------------------------------
It’s time for my yearly rant on how bad the state of the “industry” is regarding benchmarking computer hardware. I griped about this issue in the June 2004 issue (page 30). It doesn’t matter, though, as that’s not going to stop me from speaking out once again.

In terms of overall consumer technology, we’ve come a long way since then, and our advances have made our virtual rulers just that much more invaluable. We saw Intel’s Hyper-Threading truly come into its own, and more recently, both Intel and AMD have launched dual-core CPUs that are selling at price levels that are very much attainable by anyone who invests more than $900 or so in a DIY desktop computer.

If you’ve read reviews about any type of computer hardware (especially in the DIY realm but certainly not limited to it), you’ve likely been deluged with metrics of some type or another. Let me pose a question to you: Do you have a better computer if it scores a 9562, an 8.5, or a C+? Exactly. You have no clue because the benchmark number means nothing to you. I own a computer hardware review site, and for the last seven years, I have pretty much stayed away from assigning metrics. That said, there are certainly places that metrics are a necessary evil, but I think many of my editor peers have gone the way of using the fast and simple benchmark instead of truly giving the evaluation that I think you, the reader, deserves. I think you guys deserve an evaluation that truly lets you know what your computing experience would be like if using that product. Something outside of, “This product really rocks, I give it a 9.”

Most computer hardware editors come from what I like to call the “3dfx days.” It seems that most of those who didn’t come from that time learned their trade from a mentor who did. Some of you might remember those days, and some of you won’t. If you do you know that speed was king and queen. Not much else mattered. Sure, image quality and resolution were considered, but there were no editors pulling out perfectly rasterized images for comparison. Back then, all we had to do was take system A and system B and run a quick Quake II time-demo at various resolutions, and we would know real quick who was king of the hill. It was that simple. Now, here we are. Quake II won’t tell you a damn thing, but many computer hardware reviewers still use the same exact process to “review” hardware.

The timedemos that are run on many games nowadays in no way even emulate actual gameplay. An average framerate of 134fps in the latest 3D shooter game may not even be attainable. Check out all the Doom 3 benchmark numbers on the Web. You can find tons that show frame rates at 60fps or above, but the fact is that there is a frame cap present in Doom 3 that won’t even allow you to see more than 60fps. So what the hell does a reviewer accomplish showing you that a system can do 134 frames? Some will argue that they only use this metric to compare against other systems. So, one system got 134fps and the other got 121fps. Well, what does that tell me? Some editors would have you actually believe the 134fps scoring system is “better.” The fact is that benchmark will actually tell you little, if anything, about your real-world gaming experience. And if that benchmark tells you nothing about the experience you would likely have using the hardware, then what good is it?

Personal computing and gaming are no longer about frame rates. We stopped running “benchmarks” in HardOCP video card reviews in late 2003. In fact, at that time, I like to think that we stopped reviewing video cards and started evaluating the gameplay experience they provided. Processors have now gotten so robust that easy-to-produce benchmark numbers don’t tell you much about them, either. There are a lot of computer hardware editors out there who need to review their review process before they review any more hardware. Myself included. Next month, let’s talk about how it should be done instead of how it shouldn’t.

You can talk with Kyle at kyle@cpumag.com.

---------------------------------------------------------------------------------------------
That is the full article. I hope this helps the discussion a little.

After reading both sides I think both have good points and both kinds can work minus the latest article. I have read a few Hard reviews in my day and his method as far as I can tell is using "real" life benchmarks like runings fraps in a game for a certain scene and then comparing the exact thing with the competitor. The "real" being the in game and the "none real" being the synthetic benchmarks like the FurtureMark series and the Sandra benchmarks among many others.

That is to say as I understand it that the real benchmarks will give a user a better idea of what he is really going to get in that game without knowing the scores for say Aquamark 03 and how they translate into playablitiy in games with an enficious on frames per second.

Here is a good example taking from there latest review on a GT 7800 testing with BF2.

Quote: For our gameplay evaluation today, we are using the full version and playing through the single player missions in the game. Our gameplay consists of us simply playing the game as any gamer would for a period of five minutes recording the frame rate every second. Because of this, each run through will be unique so we are going to display each card separately on a graph.

Now that makes sense to me and I agree with it. It does make it much more difficult to get an acurate reading if you think about it cause there are quit a few differences that can make themselves unknown but might affect the FPS. Of course, I guess as well that can be said about runing benchmarks like 3DMark2005 as well.

Their conclusion...

Quote: With the GeForce 6800 Ultra, we found 1280x960 with 2XAA to be the highest playable quality setting. The GeForce 7800 GTX video cards allow a much better gameplay experience in Battlefield 2.

And again later of in the article...

Quote: Apples to Apples: Remember that these benchmarks do not represent real world gameplay and it might even be argued that they are misleading to a point. We highly suggest that you check out our gameplay evaluations to get a better understanding of the hardware's ability at providing you with a true gaming experience. Keeping in mind nothing is truly apples-to-apples to begin with, some people still find value in looking at performance with the resolution, AA, and AF levels being "equal." Below we are using Half-Life 2 and Need for Speed Underground 2 to look at the apples-to-apples performance.

Its such a fine line IMO. I agree with BA that biast feeling towards a review when reviewers give there experience they had is a little strange. Like BA said, if a card gave one reviewer a boner and the other did not which one is to say better...the scores of course!

I like that they use "real scores" from games in their reviews. I don't like how they smash all "none real" benchmarks cause alot of them can be used for a simple display of the performance difference. But I agree with BA and disagree with Kyle that we should take out benchmarks and give the feelings that the gear provided us.

And last thing I wanted to type here was

Quote: What Kyle is complaining about is that reviewing isn't standardized across the industry.

I like that it is not standardized like was said. It would be so boring if everyone said the same thing and ran the same exact scores. The reviews here on PR are fresh and different then the ones on Anandtech, but that does not mean either are wrong or differnet. After all, its the opinion of one person in the end. The scores are there to show the reader how he came to that. For the rest, I guess you would have the read the article.

I would like to thank all the site owners, editors, and reviewers who have emailed me their support and agreement. I especially like the stories of your own run-in's with [H]. I am glad that so many of you feel the same way we do here at PimpRig. Thank You.

Someone had to say, I'm glad it was BigAkita and PimpRig. You can't decide on a purchase based on what others simply say. Would you buy a car is someone said it gets "good gas mileage"? No, you'd want a comparison to another car. Everyone's opinions are relative to the experiences they have had. I could write a review and say the X300SE is the best card in the world, because it out performed the 8MB Savage card I was used to. I welcome Kyle's rebuttal to this, but I really think that [H]ardOCP is getting to full of itself.

Oh man, I agree with this %100, also.
Seems the money and "fame" are going to his head. I perfer numbers than boners ratings, myself.

I like CPU magazine, in fact it is my favorite computer mag.

I agree 100% with BA. I like to see how well a graphics card compares to other graphics cards.. the only way to truly do that is with numbers..

Taking away benchmarks would confuse a lot of people.

Excellent article BA.

Final ratings and the like are not what makes people decide to buy hardware, they're just used as a guide much like all the other information within the review. The end user must decide what will perform better in their machine based on all the information within all the different reviews on multiple sites. Taking in all this information gives a decent insight to how the product would perform in their own machine. 'Experience' is just worthless information compared to the technical parts. Numbers don't lie, people with 'experience' do.