StorageReview Enterprise Test Lab
We're often asked not only about our testing methodologies, but also about the environment in which we test, as enterprise storage solutions need to be evaluated in an enterprise environment that replicates what occurs in the real world. Single SSDs and HDDs can be reviewed almost anywhere, but high-end storage solutions and arrays need a more robust testing environment. That involves testing not limited to a single server, but rather to testing involving racks of servers, connectivity gear and accompanying components since overclocked PCs and DIY servers simply don't get the job done. To that end, we've developed an extensive on-site enterprise test lab to both simulate real world conditions and also offer enough compute, switching, electrical conditioning and proper environmental conditions to handle nearly all storage products. This section is an overview of the StorageReview Enterprise Test Lab, which is home to servers from almost all tier 1 providers as well as SAN and NAS appliances, PCIe flash solutions, high-speed interconnects and power conditioning products that make it possible to test the latest enterprise hardware.
The StorageReview Enterprise Test Lab includes the following components broken down into categories.
- Enterprise Storage Performance Benchmarks
- StorageReview NAS Lab Overview
- Rack Hardware
- Power / Temperature Monitoring
- TrippLite SRCOOL12K (12,000 BTU) x 2
- StorageReview Test Lab Networking Overview
- Netgear ProSecure UTM150 4 x WAN Firewall
- Netgear ProSafe GSM7352S 48-Port 1GbE Switch
- Netgear ProSafe M7100 24-Port 10GbE Switch
- Netgear ProSafe XS712T 12-port 10GbE Switch
- Netgear ProSafe GS752TS 52-Port 1GbE Switch
- Netgear ProSafe M4100 POE+ 24-Port 1GbE Switch
- Netgear ProSafe GS752TXS 52-Port 1GbE Switch x 2
- Netgear ProSafe WNDAP660 x 2
- Netgear ProSafe WNDAP620
- Mellanox SX1036 36-Port 40GigE Switch
- Mellanox SX1024 40GigE Switch
- Mellanox SX6036 36-Port 56Gb/s InfiniBand Switch
- Brocade 6510 16Gb Fibre Channel Switch
- Dell PowerEdge R720 G12 (x2)
- Dell PowerEdge R820 G12
- Dell PowerEdge FX2
- EchoStreams GridStreams High‐Density Quad‐Node Computing Platform
- HP ProLiant DL380p Gen8 (x2)
- HP ProLiant DL360p Gen8
- HP ProLiant DL360 Gen9
- HP Z620 WorkStation
- Lenovo ThinkServer RD630 (x13)
- Lenovo ThinkServer RD530 (x2)
- Lenovo ThinkServer RD240 (x4)
- Supermicro SuperWorkstation 5037A-i
- Supermicro SuperStorage Server 2027R-AR24NV (x2)
- Supermicro SuperServer 6047R-TXRF (X9DRX+-F)
- Enterprise NAS/SAN
- Storage Groups
- Toshiba MBF2600RC 600GB 10K SAS x 16
- Toshiba MK01GRRB 147GB 15K SAS x 16
- Seagate Enterprise Performance 10K.7 x 16
- Seagate Enterprise Performance SSHD x 16
- Seagate Enterprise Performance 15K.4 Turbo x 16
- Seagate Enterprise Capacity 2TB x 16
- Seagate Enterprise Capacity 6TB x 16
- Seagate Enterprise NAS 6TB x 16
- Seagate NAS 4TB x 16
- Hitachi Ultrastar 4TB 7K4000 SATA x 16
- HGST Deskstar NAS 4TB x 8
- HGST He8 8TB x 16
- HGST SSD800MM 400GB x 20
- Toshiba HK3R2 480GB x 24
- WD Xe SAS HDD x 16
- WD Red 3TB x 12
- WD Red 4TB x 8
- WD Red 6TB x 8
- WD Red Pro 4TB x 16
- WD Se 4TB x 16
- Intel SSD 520 180GB x 48
- Corsair Enterprise SSD 200GB x 48
- Protective Storage
The largest components of the StorageReview Test Lab are our two Eaton 42U S-Series racks. They allow us to put storage hardware in its native environment and have it located in close proximity to power, networking, and compute hardware. When looking at environmental needs, installing hardware in a rack-mount environment also helps with proper airflow and in keeping system and component temperatures down.
In our current layout, we dedicate one rack to permanent compute infrastructure and the other to power conditioning gear as well as more transient hardware that stays for just the length of the product review. This layout allows us to keep network cabling routing for our multiple interconnect types neatly organized, which takes up most of the space in the rear of both racks.
From the rear of each rack, you can see both power cabling and networking cables going to each system. For each server, we dual-source power from two independent circuits controlled by their own UPS. Each server group is also pre-wired to support each networking standard we work with including: 1GbE, 10/40GbE, 8/16Gb FC and 56Gb IB.
For reliable day-to-day compute and storage operation, we regulate temperature in the lab with an additional cooling unit that supplements our building's HVAC system. With our current 3000 watt operating load, we need to remove roughly 10,000 BTU from the lab which is handled by our TrippLite SRCOOL12K. As our demands increase, we can add additional cooling units until we require a larger dedicated cooling system.
At the component level, power consumption is continuously dropping as each new generation of product comes out, but at the datacenter level the total power required to run multiple switches, servers, SANs, and other required gear adds up to values well above what normal 120V circuits can handle. The StorageReview Lab offers a wide range of utility options backed with three-phase power coming into our building. In total, we have a power ceiling of 72,000 watts in our lab, providing plenty of headroom no matter what scale of projects we need to tackle.
To handle this massive power distribution on an outlet-by-outlet basis, we use Eaton and TrippLite PDUs and UPSs. The bulk of our power is fed in through our Eaton 9PX6K UPS which has a capacity of 5,400w and is balanced with a TrippLite SmartOnline that offers a 2,500w capacity. These provide dual-power to our servers to prevent single power branches from bringing down any of our primary testing equipment. For high-priority tests and core networking components, we use an Eaton 5PX with battery expansion to ride out longer power outages.
With constant 24/7 power monitoring, we make sure tests are running smoothly, power conditions are clean, and that abnormal power conditions don't affect our tests. With evaluations that take weeks or months and tests that span days individually, making sure everything is running without a blip is an important consideration when we review enterprise hardware.
To handle this massive power distribution on an outlet-by-outlet basis, we use an Eaton Advanced Monitored ePDU, Eaton 5PX and 9PX UPS, and multiple PDUs. Power is fed in through either our primary Eaton UPS with servers connected directly to it or we through our Eaton ePDU which supplies power to both of our Eaton racks. For the second rack we use auxiliary PDUs branched off the main ePDU to supply power locally.
The StorageReview Enterprise Test Lab supports nearly every interconnect standard including 1Gb, 10Gb and 40Gb Ethernet, 8Gb and 16Gb Fibre Channel, and 56Gb InfiniBand. These are supplied by partners that include Brocade, Emulex, Intel, Mellanox, Netgear and QLogic. This extensive networking infrastructure allows us to perform tests on every major storage platform while ensuring it has the fastest interface supported.
To interface with equipment over SSH, RDP, or iKVM, we use a 1GbE Ethernet infrastructure split into multiple VLANs to separate normal office and lab communications. This includes a Class C network for our office, as well as a Class B network for lab tests which gives us over 65,000 addressable hosts. Our 1GbE infrastructure is supplied by Netgear, with a broad range scaling from a fully managed L3 core switch to Smart switches for branch segments of our network. At the heart of this network is a Netgear UTM150 Firewall which manages incoming and outgoing traffic, load balances across multiple Internet connections and provides VPN access to the lab. For Internet access, we have a fiber-optic connection through Cincinnati Bell's Fioptics service.
The next step up in interconnect hardware includes both 10Gb and 40Gb Ethernet supplied by Mellanox which is routed through an SX1036 switch. This fabric is used to test SMB to large-enterprise NAS and SAN appliances that require bandwidth that can easily over-saturate standard 1GbE interfaces.
For Fibre Channel SANs and storage appliances, we have an 8Gb QLogic switch with 8/16Gb QLogic HBAs, as well as a 16Gb Brocade switch with 16Gb Emulex HBAs.
For solutions that demand even greater throughput and lower latency, we also have a 56Gb/s Mellanox InfiniBand fabric used for testing flash appliances and acting as the backbone of new application tests such as our MarkLogic NoSQL Database Benchmark.
The compute infrastructure in the StorageReview Enterprise Test Lab is divided into two sections - one for synthetic testing and the other for resource-intensive application tests. Synthetic tests require fewer system resources such as CPU and RAM, so these servers offer more modest configurations geared towards local or networked FIO testing and other duties.
In our synthetic testing environment, we have Intel E5-2600-family servers from many Tier 1 vendors such as Dell, HP and Lenovo. These include two HP ProLiant DL380p Gen8s, two Dell PowerEdge R720 G12s and five Lenovo ThinkServer RD630s. Our ThinkServer RD630s are configured with LSI 9207-8i PCIe 3.0 HBAs to ensure the highest possible bandwidth and I/O throughput when we benchmark current SATA and SAS 6.0Gb/s SSDs.
Our lab also includes past-generation servers such as the Intel-Westmere Lenovo ThinkServer RD240s which supports PCIe 2.0 connectivity. We used these servers to host VMs through VMware vSphere Hypervisor and perform enterprise tests on SATA and SAS hard drives.
The newest section of the Enterprise Test Lab is aimed at supporting our application testing environments. The primary hardware in this category comes in the form of eight Lenovo ThinkServer RD630s equipped with dual 8-core Intel E5-2650 CPUs as well as 128GB of RAM each. We also have a quad-node EchoStreams GridStreams server with each node equipped with dual 6-core Intel E5-2640 CPUs and 64GB of RAM each. These servers offer the compute resources necessary to consume the high I/O that current-generation storage devices can dish out.
Outside of our primary Lenovo compute cluster, we also work with manufacturers such as HP, Dell, and Super Micro Computer with their newest Romley servers as we evaluate new testing platforms or approach product reviews that are unique to a specific manufacturer's hardware. With a growing lab of workstation and rack-mount compute hardware, we are equipped to handle nearly all enterprise storage products.
As the lab grows and evolves we'll be sure to update this post with the latest gear and more detail on how we're using it.