StorageReview.com's Testbed3

Introduction When we initially opened up shop here at StorageReview.com, our "testbed," the constant machine that we center our hard drive reviews around, was a 266 MHz Pentium II on 440LX chipset equipped with 64 MB of PC66 SDRAM. At that time (1Q 1998), the 333 MHz P2 was just hitting the streets as the premiere desktop CPU. ATA-33 and Ultra2 SCSI were new standards... we were blissfully on our way. This machine, affectionately called "Hoss" due to its incredibly heavy (and ridiculously expensive) steel case, carried us through 1998 and much of the next year. As autumn '99 approached, however, it became clear that "Testbed1" was getting a bit long in the tooth. Since we prefer to change as many variables as possible when a single switch is necessary, we decided to hold off upgrading the hardware in anticipation of three key elements: the OS that was to merge Microsoft's Windows 9x and NT series (Windows 2000), Intel's i820 chipset (the first to feature Intel's ATA-66 controller) and WinBench 2000. Our projected deployment was set for late 1999.

Things didn't turn out quite the way we planned. In addition to a February 2000 delay, Win2k ended up being an NT-successor rather than the unifier of kernels as Redmond originally touted. The i820 chipset, designed from the ground up to use outrageously expensive RDRAM memory, was incredibly buggy. And perhaps worst of all, there was no WinBench 2000. These setbacks delayed the introduction of our second-generation platform. Hushed whispers spoke of SR being too set in its ways to upgrade, that SR didn't care and was too lazy to bother. True? No way! As it became increasingly clear that WinBench 2000 was non-existent and that the i820 would never mature into a viable platform, we decided to center Testbed2's release on Windows 2000's commercial debut. The site's second anniversary, March 12 2000, set a perfect target.

This second-generation testbed, dubbed "Millennium," was built on a 700 MHz Pentium III. Oh sure, to counter AMD's impending 1 GHz Athlon, Intel had "released" the 1 GHz P3 around then, but it was no where to be seen... nor would it be for half a year. Even the 800 MHz part, ostensibly available, was in short supply. We paid dearly for 700 MHz, the best we could obtain at the time while still prepping for a March 12 release. This testbed was supported by a 440BX based motherboard and 128 MB of PC100 SDRAM. The BX, unfortunately, remained equipped with a southbridge that only supported ATA-33 standards. Thus, we turned to Promise's Ultra66, the most popular add-on solution, for our reviews. On the SCSI side of things, an upgrade to Adaptec's Ultra160 card kept us rolling.

A more significant change, however, arrived in our software and methodology. Windows 2000, while not the unified OS it was hyped to be, represented the future of PC computing. To streamline tests and maintain consistency we chose to disregard Windows 98 SE and the impending Windows ME. Since we (as well as many readers) felt that the aging, industry-standard WinBench 99 was getting poorer at assessing hard drive performance, we set out with an additional tool: Intel's IOMeter.

We regarded TB2 as something that would hopefully carry us for another two full years after which, centered on SR's fourth anniversary, we would debut the third-generation testbed... or testbeds. The name "Testbed3" carried a dual meaning. It would not only be the third successive hard disk platform but would also be the time where we deployed identical configurations and installations to all our reviewers (along with myself in hard disks, associate editors Tim Zakharov and Terry Baranski later joined us to cover CD/CDR/DVD and controller performance respectively).

Setup-wise, a platform upgrade again hinged upon several key features. The first, of course, was the release of MS's next OS, Windows XP... this time (really!) purported to unify their consumer and professional kernels. Second was the availability of consumer level motherboards that supported 64-bit cards and perhaps 66 MHz operation. Finally, the ever-tantalizing Serial ATA and Ultra320 interface upgrades made up criterion number three.

About eight months ago, right around SR's third anniversary, we embarked on an extended research project. What factors are involved in hard drive speed? Were the requirements for optimum desktop and server performance more similar to or more different from each other? Were SR's current tools delivering dependable performance predictions? We (as well as many readers) believed we already had the answers to this... and boy, were we off. Let us state right up front that we've gained a tremendous amount of knowledge in hard drive performance assessment over these past eight months. In the interests of truth, objectivity, and accuracy, we've thrown aside pride, preconceptions, and biases to get to the bottom of drive performance. We hope readers will be just as willing to accept new measurement ideas founded in the pool of evidence that follows later in this article.

As our experimental research progressed, it became clear that we had a duty to revise our methodologies even if the hardware and OS to support Testbed3 wasn't quite there yet. We hatched a plan for "Testbed 2.5" where we'd refresh our methodologies while keeping hardware and possibly the OS the same. As it would be, however, fate intervened. A month ago, the imaged software installation that we came to depend on over the course of more than 1.5 years became corrupted. That image contained updates, driver revisions, etc. that were no longer available and thus contained a software installation that could no longer be replicated. Then, incredibly, while flashing the machine's motherboard in preparation for a Testbed 2.5 update, TB2's power supply failed, leaving the motherboard's flash ROM in an unsalvageable state. The message is clear: The future is now. It's time for Testbed3!

Due to both financial realities and industry delays, TB3 is not the originally-intended unified platform deployed across three different reviewers. Rather, it represents a necessary and overdue migration to a new hard drive test platform. Revised optical and controller testbeds are to follow in the near future.

We're very proud of the evaluation suite that we've assembled for Testbed3. The research, methodology, design, and programs that we're debuting represent over 1000 man-hours of effort between just two individuals. As you read over the changes we've made, we hope you'll agree that SR and readers alike are destined for a new era in hard drive judgment.

Oh, the name of our third-generation Testbed -- "Renaissance."