Testbed4

A Brief History of StorageReview's Testbeds When we launched StorageReview back in 1998, one of our principal goals was to maintain a consistent, unchanging test platform that would enable readers to directly compare a wide variety of drives with each other. Back then, when one could find them at all, hard drive reviews were always conducted on "the latest and greatest" machine that the individual reviewer could put together. Though such articles occasionally featured one or two other drives tested in the same machine for comparison, by and large, it was difficult to directly compare contemporary drives with one another. This changed, however, with StorageReview's debut of Testbed1. Testbed1: Our initial testbed was a 440LX-based 266 MHz Pentium II machine featuring an ATA-33 controller operating off of the PIIX4 southbridge. An Adaptec 2940U2W provided Ultra2 (80 MB/sec) SCSI functionality. Windows 95 and NT 4.0 laid the foundation for our tests, ZD's WinBench 98 and Adaptec's ThreadMark 2.0.

Despite its flaws, WinBench 98 truly was the best tool to measure single-user disk performance at the time. ThreadMark was our initial attempt to present multi-user performance. Looking back, however, its clear that the benchmark came up short.

Testbed1 nonetheless carried us through dozens of drive reviews spanning two years from March 1998 to March 2000. By then, it was clear that both the hardware and benchmarks required updating.

Seagate, Maxtor, and Western Digital have all entered the fray with SATA units specifically tuned for the enterprise sector. While leveraged from consumer-class SATA designs, these differentiated models undergo tests under different workloads, often enjoy longer factory burn-in cycles, are rated for longer mean times between failures, and are backed by a more business-oriented 5-year warranty.

Let us take a closer look at three 500-gigabyte units that squarely aim to seize the burgeoning nearline enterprise sector where cost and capacity rather than sheer IOps drive the market.

Testbed2: Testbed2 initially hinged upon three key factors. First was the impending introduction of Windows 2000, which, at the time, was heralded as the release that would unify Microsoft's consumer (Win9x) and professional (NT) kernels. Next was Intel's i820 chipset, the first chipset that would introduce Rambus memory, the RAM of the future. Lastly, Testbed2 was to take advantage of ZD's WinBench 2000 to update our single-user performance tests.

All three of these updates failed to materialize. Microsoft decided it needed more time to move their consumer operating system to the NT core and instead updated Windows 95 yet again in the form of Me. The i820 chipset suffered from delays and bugs... and Rambus memory, of course, never took off. Finally, it turned out that WinBench 99 was the last iteration of ZD's venerable component-level benchmark.

Thus, we stuck with a 700 MHz Pentium III paired with Intel's tried-and-true 440BX chipset. Promise's Ultra66 provided ATA-66 operability while Adaptec's 29160 delivered Ultra160 SCSI compatibility. We chose to go with Windows 2000 Professional and abandoned the Win9x core entirely since the former paired with the NTFS file system represented the future of desktop machines.

Though we presented WinBench 99 results on Testbed2, our big focus was on testing with IOMeter. On the surface, IOMeter's highly-customizable nature seemed promising- tinkering with its settings yielded a pattern that we dubbed "workstation," one that we believed could best represent single-user performance. Unfortunately, we were dead wrong. IOMeter lacks any facilities to simulate localized drive access- i.e., the tendancy for a given piece of required data to be very close to the last piece of data accessed. This delivered "workstation" numbers that differed very little from the server results returned by IOMeter. Though Testbed2 was originally mapped out for a two-year run, it became painfully clear that our methodologies were flawed and that an update was needed as soon as possible. Hence, Testbed2 lasted just 19 months.

Testbed3: Testbed3 was truly a massive undertaking, the fruition of nearly one-thousand hours of research into developing the ideal way to assess single-user performance. The hardware consisted of an i850-based Pentium 4 featuring Promise ATA and SATA controllers and Adaptec's 29160 SCSI controller. The most significant change, however, was on the software side. Microsoft finally released Windows XP, the version that finally unified both its OS lines into a single whole. More importantly, Intel's IPEAK SPT 3.0 provided us with an opportunity to capture and exactingly reproduce the accesses generated by any Windows-based application. IOMeter, which was always a great tool for assessing multi-user performance, remained for our server-side tests.

We thus segued from a "dark age" of sorts where Testbed2 presented inadequate and downright misleading results to a renaissance with Testbed3- IPEAK in effect allowed us to design our own custom benchmarks, the StorageReview Desktop DriveMarks. Testbed3 debuted in November of 2001 and provided a stable, unchanging platform for nearly four years. It was retired this summer.

This brings us to Testbed4. Unlike our three previous changes, Testbed4 is far from revolutionary. When we moved from Testbed1 to Testbed2 and from Testbed2 to Testbed3, we were motivated primarily by concerns that our methodologies were yielding inaccurate results. Testbed3, however, has stood the test of time- it is only the dated hardware and captures of aging applications that have finally driven us to move on. Testbed4 is an evolutionary update- IPEAK SPT and IOMeter remain as our cornerstone benchmarks. We recommend that readers re-read Testbed3's introduction for extensive information and discussion of IPEAK SPT and its superior ability to assess single-user drive performance.