Reviews Leaderboard Database Reference Search StorageReview Discussion Reliability Survey Search About StorageReview.com Contents

Same Drives = Same Performance?
  January 21, 2001 Author: Terry Baranski  
Special thanks to Promise Technology, Inc. for providing the evaluation units.


Introduction

Just how much performance variation exists between multiple samples of a given drive? This question pops up every so often in the StorageReview.com Discussion Forums as well as in usenet newsgroups. Some say that differences in performance can be significant, while others claim that manufacturers adhere to strict performance tolerances when shipping drives (e.g., seek time must be within 0.05ms of spec for the drive to be considered shippable).

As our readers know, StorageReview.com uses Ziff-Davis' WinBench 99 to measure access time. From access time, seek time can easily be calculated by subtracting the drive's average rotational latency. It is often interesting to compare measured seek times to the figures specified by the manufacturer. When the seek time derived from WinBench's reported access time differs significantly from the manufacturer's claim, there are three possibilities: 1) the manufacturer's specified seek time is either too high or too low, 2) WinBench is wrong, or 3) the particular drive tested, for whatever reason, has an out-of-spec seek time.

How does one decide which case is correct? In order to arrive at an accurate conclusion, multiple samples of the drive in question must be tested. For the most accurate results, the drives should also be of the same capacity. Unfortunately, it is not often that StorageReview.com has a chance to do this; we generally receive only one sample for review.


Perfect Timing?

A short while ago, I was approached by StorageReview.com regarding RAID reviews. Since I'm extremely interested in hard drives and related technology, I immediately jumped at the opportunity to join the SR team. Since then, I've been putting a few cards through their paces... and dealing with the inevitable issues that arise in IDE RAID (and perhaps RAID in general).

Promise Technology was kind enough to send us four Maxtor DiamondMax 80 drives along with their RAID adapters. A few weeks ago, the issue of performance differences between multiple samples of a given drive arose. These four DiamondMax 80's give us a perfect opportunity to shed some light on the issue. With that in mind, I began testing the four samples in both Winbench and IOMeter. Would all four drives perform about the same? Or would there be significant performance variations between two or more drives? We'll find out shortly, but first, let's outline the testbed.


Testbed Specs

  • Operating System: Windows 2000
  • Motherboard: Abit SL6, BIOS revision SX (10/26/00)
  • Processor: Celeron 600
  • RAM: 128MB Generic PC100
  • Floppy Drive: 1.44MB
  • Boot Drive: Maxtor DiamondMax Plus 6800 13GB (single NTFS partition)
  • RAID Drives: Four Maxtor Diamondmax 80's (80GB)
  • CD-ROM: Lite-On 32X
  • ATA Controller: Intel 82801AA (DM+ 6800 on primary channel, CD-ROM on secondary channel)
  • Drive Cooler: PC Power & Cooling Bay-Cool III
  • Display Adapter: Intel 82815 Graphics Memory Controller Hub (GMCH)
  • Network Card: Intel EtherExpress Pro 100 (Only used for downloading drivers, BIOS updates, and applications. It is removed from the system prior to any benchmarking.)
  • Keyboard: Micro Innovations Standard Keyboard
  • Mouse: Logitech Wheel Mouse
  • Case: Enlight Endura EN-7237
  • Power Supply: Enermax EG351P-VE 330W


RAID OS, Drivers, and Benchmarks

The first step in testbed configuration was OS (Windows 2000) installation. The procedure was straightforward -- a single 13GB NTFS partition was created on the DiamondMax Plus 6800 after booting from the Win2000 CD. The rest of the installation went as smoothly; nothing more than your run-of-the-mill OS installation.

After the installation was complete, the latest drivers were installed for the Intel EtherExpress Pro 100, and the Intel i815 chipset (including graphics and storage drivers). The motherboard BIOS was flashed to the most recent available at the time, version SX. The Promise Ultra66's BIOS was also flashed to the latest available version, v2.0. Due to issues we had with the then-current Ultra66 drivers, we stuck with the drivers inherent in Win2000.

Next, screen resolution was set to 800X600, with refresh rate at 85Hz. Auto-insert notification was disabled via RegEdit. Through Windows Update, Windows 2000 Service Pack 1 and the current Windows 2000 Critical Updates Package were both installed.

Then the following benchmarks were installed: ZD's Winbench 99 v1.1, Intel's IOMeter v1999.10.20, and TestaCD Lab's HDTach v2.61. WinBench and IOMeter results will be formally presented in reviews.

Finally, the boot drive was defragmented using Windows 2000's built-in utility.

RAID cards will be evaluated in PCI slot 1 (next to the AGP slot). Because the SL6 has integrated video and sound, all other PCI slots (as well as the AGP slot) are empty during benchmarking. Three of the four DiamondMax 80's were attached to the drive cooler, with the final unit in a standard 3.5 inch bay.


And now, for the DiamondMax 80 tests...

With all of that settled, we can return to the topic of this article: performance variation between multiple samples of the same drive. The following benchmarks were run with the boot drive on the primary channel of the motherboard's ATA controller, the CD-ROM on the secondary channel, and each DiamondMax 80 on the primary channel of the Promise Ultra 66 (in PCI slot 1) during testing.


WinBench Results

The following WinBench tests were run five times on each drive: Disk/Read Transfer Rate, Disk CPU utilization, Disk Access Time, Business Disk WinMark 99, and High-End Disk WinMark 99. A single run of each of the above tests was considered a "trial", with five trials being conducted for each drive. The machine was rebooted between trials. Each test's final score represents the average of the five runs.


Sustained Transfer Rate Graphs:

STR Graphs Of Each Sample

Drive A

Drive B

Drive C

Drive D
- Click thumbnail for larger image -

One can see that all four drives have a maximum transfer rate of just under 30 MB/sec and a minimum transfer rate of around 17.5 MB/sec. This was expected: different samples of the same drive should, of course, have the same number of sectors per track in any given zone; therefore, sequential transfer rates should be identical.

Two graphs stand out. Note that drive C's chart is quite jagged in the first four zones - zone 1 in particular. This behavior was repeatable and consistent, and we're somewhat at a loss for an explanation. Drive D's graph is also interesting: there appears to be at least one remapped sector about halfway through the first zone.


The rest of the WinBench scores...

Ziff Davis WinBench 99 under Windows 2000 Professional using NTFS
Benchmark Drive ADrive BDrive CDrive D
Business Disk WinMark 99 (KB/sec) 6556 6588 6586 6324
High-End Disk WinMark 99 (KB/sec) 16420 16360 16240 16320
AVS/Express 3.4 (KB/sec)16520 15820 15680 16000
FrontPage 98 (KB/sec)63640 63680 63000 64100
MicroStation SE (KB/sec)21200 20960 20880 20900
Photoshop 4.0 (KB/sec)8616 8708 8566 8616
Premiere 4.2 (KB/sec)14240 14040 14220 13940
Sound Forge 4.0 (KB/sec)17980 17960 18860 18220
Visual C++ (KB/sec)16500 16920 16640 16620
Disk/Read Transfer RateStorageReview.com
Beginning (KB/sec)29767 29800 29443 28900
End (KB/sec)17500 17500 17500 17467
Disk Access Time (ms)15.04 15.16 15.26 15.18
Disk CPU Utilization (%)2.72 2.74 2.73 2.74

One can see that drives A, B, and C turn in virtually identical Business Disk WinMark 99 scores. Drive D, on the other hand, lags the others by 3%. High-End Disk WinMark 99 scores exhibit little deviation- all drives scored within 1% of each other. It should be noted, however, that there are some significant differences between drives in the individual application tests that make up the High-End score. For example, drive C's Sound Forge score is about 3.7% higher than both drive A and drive B's scores. However, it's important to note that these individual application tests run very quickly, so there are bound to be some differences in the scores, even when doing multiple runs on the same drive. Case in point: the difference between the lowest and highest of the five Sound Forge trials approaches 10%. As a result, we believe that the differences in application scores between drives are a product of the benchmark itself rather than inter-sample performance differences.

There is a 0.22ms difference between the lowest and highest access time scores (15.04ms for drive A, and 15.26ms for drive C). Does this mean that drive A seeks faster than the other three drives? While this is certainly possible, we have some reservations about drawing such a conclusion. One concern that we have with WinBench 99's access time test is its relatively short run-length. When measuring something such as average access time, accuracy increases as the number of strokes increases. The logic is obvious: when calculating an average, increased samples reduce the effect of outliers and other undesired effects.

Another concern we have about WinBench's access time test is a phenomenon that we observed while testing of these units. The table below consists of five access time scores for each drive (corresponding to the five trials that were run on each drive):

Access Time using Ziff Davis WinBench 99 under Windows 2000 Professional using NTFS
Access TimeDrive ADrive BDrive CDrive D
Trial 1 15.2 ms 15.4 ms 15.4 ms 15.3 ms
Trial 2 15.0 ms 15.1 ms 15.3 ms 15.2 ms
Trial 3 15.0 ms 15.1 ms 15.2 ms 15.2 ms
Trial 4 15.0 ms 15.1 ms 15.2 ms 15.1 ms
Trial 5 15.0 ms 15.1 ms 15.2 ms 15.1 ms

Note that, for all four drives, WinBench's reported access time decreases as more trials are run (e.g., the access time reported in trial 5 is always lower than the access time reported in trial 1). This behavior is very consistent. One explanation that occurrs to us was the fact that the drives weren't formatted in between trials. However, the question arises: why should they be? An access time measure purports to read random sectors, so reformatting shouldn't affect scores. A reboot was inserted between trials, so caching may be eliminated. We're at a loss to explain this issue, which is the main reason why it concerns us...

Finally, CPU utilization scores are virtually the same for all four drives, as was expected.


IOMeter results

IOMeter - File Server Access Pattern - Load = Linear
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 65.94 0.71 15.16 ms 0.60 % 109.90
Drive B 65.78 0.71 15.20 ms 0.57 % 115.40
Drive C 65.57 0.72 15.25 ms 0.66 % 99.35
Drive D 65.97 0.72 15.15 ms 0.63 % 104.71

IOMeter - File Server Access Pattern - Load = Very Light
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 68.42 0.75 58.46 ms 0.71 % 96.37
Drive B 68.65 0.74 58.26 ms 0.60 % 114.42
Drive C 68.07 0.75 58.75 ms 0.66 % 103.14
Drive D 68.15 0.73 58.69 ms 0.65 % 104.85

IOMeter - File Server Access Pattern - Load = Light
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 79.64 0.87 200.86 ms 0.79 % 100.81
Drive B 79.23 0.86 201.86 ms 0.73 % 108.53
Drive C 78.54 0.85 203.65 ms 0.80 % 98.17
Drive D 78.49 0.85 203.82 ms 0.73 % 107.52

IOMeter - File Server Access Pattern - Load = Moderate
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 89.25 0.96 716.68 ms 0.96 % 92.97
Drive B 89.00 0.98 718.76 ms 0.90 % 98.89
Drive C 88.08 0.95 726.25 ms 0.95 % 92.72
Drive D 88.29 0.95 724.60 ms 1.00 % 88.29

IOMeter - File Server Access Pattern - Load = Heavy
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 98.65 1.07 2591.71 ms 1.35 % 73.07
Drive B 98.09 1.06 2606.49 ms 1.31 % 74.88
Drive C 97.25 1.05 2626.84 ms 1.28 % 75.98
Drive D 97.43 1.06 2622.30 ms 1.35 % 72.17

IOMeter - Workstation Access Pattern - Load = Linear
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 77.36 0.60 12.92 ms 0.83 % 93.20
Drive B 76.73 0.60 13.03 ms 0.83 % 92.45
Drive C 76.48 0.60 13.07 ms 0.71 % 107.72
Drive D 76.37 0.60 13.09 ms 0.74 % 103.20

IOMeter - Workstation Access Pattern - Load = Very Light
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 79.84 0.62 50.10 ms 0.80 % 99.80
Drive B 79.19 0.62 50.50 ms 0.74 % 107.01
Drive C 78.50 0.61 50.95 ms 0.74 % 106.08
Drive D 78.52 0.61 50.94 ms 0.79 % 99.39

IOMeter - Workstation Access Pattern - Load = Light
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 91.75 0.72 174.34 ms 1.00 % 91.75
Drive B 91.44 0.71 174.96 ms 0.88 % 103.91
Drive C 89.78 0.70 178.19 ms 0.95 % 94.51
Drive D 90.26 0.71 177.22 ms 0.91 % 99.19

IOMeter - Workstation Access Pattern - Load = Moderate
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 102.58 0.80 623.55 ms 1.13 % 90.78
Drive B 101.51 0.79 630.15 ms 1.02 % 99.52
Drive C 100.63 0.79 635.77 ms 0.98 % 102.68
Drive D 100.95 0.79 633.68 ms 1.03 % 98.01

IOMeter - Workstation Access Pattern - Load = Heavy
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 111.90 0.87 2284.06 ms 1.54 % 72.66
Drive B 111.77 0.87 2286.81 ms 1.59 % 70.30
Drive C 110.31 0.86 2317.64 ms 1.56 % 70.71
Drive D 111.07 0.87 2301.64 ms 1.49 % 74.54

IOMeter - Database Access Pattern - Load = Linear
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 70.68 0.55 14.14 ms 0.74 % 95.51
Drive B 70.11 0.55 14.26 ms 0.66 % 106.23
Drive C 69.75 0.54 14.33 ms 0.70 % 99.64
Drive D 69.47 0.54 14.39 ms 0.68 % 102.16

IOMeter - Database Access Pattern - Load = Very Light
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 72.35 0.57 55.28 ms 0.75 % 96.47
Drive B 72.04 0.56 55.52 ms 0.73 % 98.68
Drive C 71.02 0.55 56.31 ms 0.66 % 107.61
Drive D 71.26 0.56 56.13 ms 0.69 % 103.28

IOMeter - Database Access Pattern - Load = Light
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 82.40 0.64 194.13 ms 0.85 % 96.94
Drive B 82.51 0.64 193.88 ms 0.78 % 105.78
Drive C 81.47 0.64 196.37 ms 0.79 % 103.13
Drive D 80.76 0.63 198.07 ms 0.85 % 95.01

IOMeter - Database Access Pattern - Load = Moderate
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 91.47 0.71 669.39 ms 0.99 % 92.39
Drive B 90.90 0.71 703.72 ms 0.97 % 93.71
Drive C 89.93 0.70 711.24 ms 0.91 % 98.82
Drive D 89.99 0.70 710.95 ms 1.01 % 89.10

IOMeter - Database Access Pattern - Load = Heavy
IOMeter Tests IO/sec MB/sec Response Time CPU Util. IO/CPU%
Drive A 99.44 0.78 2570.32 ms 1.40 % 71.03
Drive B 98.70 0.77 2588.35 ms 1.38 % 71.52
Drive C 98.05 0.77 2606.30 ms 1.35 % 72.63
Drive D 97.78 0.76 2612.42 ms 1.38 % 70.86

The first thing that stands out about the above IOMeter scores is that all drives score very, very closely to one another in each of the 15 tests. In fact, the average difference between the highest and lowest score for any given test is only 1.33 IO/sec. But is 1.33 IO/sec within IOMeter's margin of error, or is it an indication that the drives are performing differently from one another? Attempting to answer this question, we ran IOMeter on one of the DM 80's four times. Once these four runs were completed, we calculated the average difference between the highest and lowest score for each test. In this case, the result was 0.63 IO/sec. This indicates that the differences in scores we're seeing with these four drives are not entirely a product of the benchmark itself.

With this in mind, it's interesting to take a look at which drives are performing better, and which ones are performing worse. As some may know, SR's IOMeter tests consist of a total of 15 runs: 3 access patterns, with 5 variations of "outstanding IOs per second" in each one. The table below shows how many times each drive finished a test with a given rank (1st place, 2nd place, etc):

IOMeter Rank Analysis
IOMeter RankDrive ADrive BDrive CDrive D
First Place 12 2 0 1
Second Place 3 12 0 0
Third Place 0 1 5 9
Fourth Place 0 0 10 5

This table reveals some very interesting figures. For example, drive A presents the best score in 12 out of the 15 IOMeter tests. In the other 3 tests, turns in the second-best score. Drive B gets the silver metal by managing the second-place in 12 out of the 15 tests. Drives C and D, however, both finished in either 3rd or 4th place in all but 1 test.

So what do these figures mean, if anything? Due to the consistency between various IOMeter tests, it is our belief that they indicate very, very slight differences in performance between the four drives. It's impossible to pinpoint the cause of these small differences, but it seems likely that slight differences in average seek time are responsible. How much of a difference is required to cause these small variations in IOMeter scores? We saw above that WinBench reports small differences in access time, yet we've explained why we're a bit wary of placing our complete trust in those results. Wouldn't it be nice if we could test access time with IOMeter as well?

Although the IOMeter scores above include an "average IO response time" category, seek time cannot be accurately determined from this metric since all of SR's IOMeter access patterns include a certain percentage of writes and/or sequential IO's. Therefore, we ran the following IOMeter access pattern on each of the four drives:

IOMeter Access Time Specification
Transfer Size Request 512 bytes
% Reads 100%
% Random 100%

The test was run with just one outstanding IO to eliminate the effects of IO reordering. (When commands are reordered for optimal head movement, average seek time cannot be measured since average seek distance tends to decrease as the number of outstanding commands increases.) Like all of SR's IOMeter tests, this pattern was run for 10 minutes with a 30 second ramp-up time. The test was run on each drive twice. The average results:

IOMeter Access Time Test
IOMeter TestsDrive ADrive BDrive CDrive D
Total I/Os per Second 64.97 64.79 64.25 64.53
Total MBs per Second 0.0317 0.0316 0.0314 0.0315
Average I/O Response Time (ms) 15.38 15.43 15.55 15.49
Maximum I/O Response Time (ms) 31.88 46.58 47.49 46.06
% CPU Utilization (total) 0.61 0.58 0.54 0.56
CPU Effectiveness (IO/%CPU) 105.61 110.95 117.37 113.31

The important field is "Average IO Response Time", which represents the drive's average access time in this particular test. The above results are consistent with the previous IOMeter scores, as well as with WinBench access time scores. Note that the order in which the drives finished (from best to worst) in this test is A-B-D-C- exactly the same order the drives finished the WinBench access time test. Also, looking back at the IOMeter rank table, one again sees the same A-B-D-C pattern.

It is, however, interesting to note that the Average IO Response Times shown above - despite showing the same A-B-D-C pattern as the WinBench access time scores -- are consistently higher by about 0.30ms on average. The cause? There are two known differences between these tests: 1) IOMeter's test runs for a much longer period of time, and 2) IOMeter tests are run on unformatted drives, while all WinBench tests must be run on a drive that is formatted. Excluding these differences, the tests should be the same - they both presumably read random sectors, and as a result test access time in the same way. We fail to see how the 2nd difference would affect the scores; however, we mentioned earlier that we have more faith in a test that takes longer to run, so it's very possible that reason number 1 is at least partially responsible for the differences.


Conclusion

The benchmarks presented above strongly indicate that there are very slight performance differences between the four tested drives. We theorized that miniscule differences in average seek time were responsible for this; the IOMeter access time backs up this theory. The IOMeter test also sheds some light on just how much seek time difference is required to produce these small differences, as well as serving as a "consistency check" - verifying that the results correlated with other tests in both IOMeter and WinBench.

It's important to point out that the performance differences observed in these trials are minute at best. Such differences certainly aren't noticeable to any human being, nor are they detectable by the vast majority of benchmarks (the Business WinMark scores don't reflect our findings, and it could be argued that the A-B-D-C pattern of the High-End scores is coincidental since only 1 of the 7 High-End subtests follows this pattern). Therefore, a difference of 1 or 2 IOs/sec in IOMeter is hardly significant in our opinion; in fact, we initially assumed that these differences were caused by IOMeter itself. Further investigation, however, reveals that there is indeed a pattern to these differences. IOMeter access time tests reveal the same pattern, strengthening the original results.

These findings should come as no surprise. After all, hard drives are mechanical devices with physically moving parts. Very small differences in performance should be expected; it would be unreasonable to expect each and every drive to have the exact same seek speed.

I'm pleased that I had the chance to examine four drives for performance differences. It was an issue that I had thought about in the past, but never had a chance to test on my own. That being said, I hope you have enjoyed my first article. Comments are welcome -- via email, or the discussion forums.


HOME | ARTICLES | LEADERBOARD | PERFORMANCE DATABASE | REFERENCE GUIDE
COMMUNITY | RELIABILITY SURVEY | SUPPORT SR! | ABOUT SR |

Copyright © 1998-2005 StorageReview.com, Inc. All rights reserved.
Write: Webmaster