Two years ago in the hardware-enthusiast community, we noticed a trend. New sites were constantly being formed, joining established veterans such as TomsHardware.com and Anandtech.com by covering "glamorous" hardware topics such as 3d acceleration, CPUs, and motherboards. Even as unfathomable numbers of hardware sites continued to start up, topics such as hard drives and storage seemed sadly neglected- to those startups, hard drives were boring! Everyone wanted a piece of that video card pie. By January 1998, we decided to do something about it. Thus StorageReview.com was born.
We decided to set forth on reviews of equipment that most sites regarded as unworthy of attention. In doing, so, however, we wanted to make sure it was done right. Far too often we witnessed performance judgments being made on a particular piece of hardware as it sat in the reviewer's personal system. Often he would compare the new piece with an older one, one that was tested with, say, a slower CPU. Such things simply didn't sit well with us. We wanted to ensure that there would be no other external influences when we found one particular drive to be faster than another. This meant we had to keep a constant, dedicated testbed.
In February 1998, we commenced with tests on no less than 14 hard disks from 6 unique manufacturers. Our test system, specifically built for the purpose, was pretty contemporary at the time. We chose to go with an LX motherboard (the most current chipset available at the time) outfitted with a 266 MHz Pentium II (the "Deschutes," 333 MHz P2 was just starting to trickle into general availability). 64 megs of (PC66) SDRAM, a Matrox Millennium II, and, for SCSI drives, an Adaptec AHA-2940U2W housed in a monstrous PC Power & Cooling Solid Steel tower rounded out the important specs. We decided that tests should be run on completely empty drives. As a result, our system housed two boot drives: A Western Digital Caviar AC31600 for ATA configurations and a Seagate Hawk 4XL for SCSI units.
To keep comparisons between drives fully valid, we pledged to keep our testbed hardware consistent until the equipment was truly out of date and unrepresentative of our readers. As time passed, we did our best to adhere our pledge. July of 1998 saw our AC31600 boot drive fail; we replaced it with a 4.3 gig Maxtor DiamondMax 2880. The advent of ATA-66 drives caused us to add a Promise Ultra66. Other than these two components, the testbed has indeed remained unchanged through the years.
By the Summer of 1999 we were hearing from users who, while understanding our desire to deliver consistent test results, also believed that our test system was getting a bit elderly. We concurred. At the time, however, it seemed to be quite foolish to attempt an upgrade. Intel's next chipset (most importantly including native ATA-66 support) was "just around the corner." Surely, we thought, waiting for the next big Intel chipset upgrade, one which everyone would use, would be wise. Microsoft's next-generation operating system was also imminent. A strong contingent of readers argued that Windows 2000 rather than Windows NT 4.0 should anchor our new suite of benchmarks. And speaking of BenchMarks, Ziff Davis' Benchmark Operation, who's software we used as our primary measure of disk performance, traditionally released new versions of its programs in the fall. Since we already endured a switch from WinBench 98 to WinBench 99, we decided that the change from WinBench 99 to, presumably, WinBench 2000 should occur while switching out all other variables.
As it turned out, everything worked against us. The i820 chipset was plagued with bugs that haven't been fully resolved to this day. The extremely high-cost of RDRAM and poor performance of an i820/SDRAM combo has kept the chipset out of the hands of the few who would have otherwise been willing to try it. Windows 2000 missed its mid-Fall debut, slipping to early 2000. And Ziff Davis decided that there wouldn't be a WinBench 2000.
By the end of last year many readers were openly critical about the aging testbed (a monolith we call "Hoss"). Despite our explanations, we witnessed accusations everywhere that we were too "lazy to change," too "set in our ways." We decided we had to act quickly- we couldn't afford to wait for all three variables (chipset, OS, and benchmarks) to settle down. The decision therefore was to hinge our migration on the official release of Win2k. Thankfully, despite the OS's 63,000 known issues, Microsoft went through with their target date of Feb 17th 2000. Copies of the OS were available from certain channels several weeks in advance. By the beginning of Feb, it was clear that the time to move had arrived.
Our plan was to assemble a new testbed (new in all aspects- hardware, OS, benchmarks and name - we call this one "Millennium") and launch with a retesting of all drives we currently possess. We would then combine this with a transitory period during which newly reviewed drives are tested in both environments, hopefully resulting in a smooth and valid transition to the new setup without creating a vacuum of comparison that wouldn't be filled for months.
What We Chose and Why...