August 30th, 2015 by Kevin OBrien
In the Lab: Supermicro 1028U-TNR4T+ Lab Integration
After reviewing the Supermicro SuperServer 1028U-TNR4T+, we've added it into our test lab in a role hosting VMware vCenter as well as other VMs that keep tests working smoothly in the background. As part of this integration, the SuperServer has received some upgrades and add-ons, including an Emulex 16Gb FC card as well as an LSI/Avago 9361-8i SAS3 RAID card. The RAID card gives us support for SAS devices connected inside the front-mount drive bays, which the platform ships enabled for from the factory. The Emulex 16Gb FC card helps us connect to traditional SAN components in our lab, whereas the onboard 4 x 10GbE handles vMotion and iSCSI storage connectivity. In this In The Lab piece we'll cover the upgrade steps as well as the background on why this server is well suited for a virtualized environment.
Supermicro SuperServer 1028U-TNR4T+ Configuration:
- Dual Intel E5-2699 v3 CPUs (2.3GHz, 18-cores, 45MB Cache)
- 384GB RAM (16GB x 24 DDR4)
- LSI 9361-8i SAS3 RAID Card
- 8 x Seagate 10K.8 1.8TB 10K SAS w/ TurboBoost in RAID6
- 1 x Emulex 16GB single-port FC HBA
- VMware ESXi vSphere 6.0 / Enterprise Plus
What makes the Supermicro SuperServer 1028U-TNR4T+ unique is its twin NVMe bays on the front, which when combined with 8 additional SAS3-capable bays makes it a very compelling system for high-performance storage in a 1U footprint. In our lab we leverage a wide range of storage configurations for systems used to test other devices as well as systems used to keep those tests going. As we've streamlined gear in our brand new enterprise test lab, we've looked for areas to upgrade our primary infrastructure. Packed full of DRAM, with strong CPUs and high-performance storage, the 1028U-TNR4T+ allows us to contract powered-on servers to a single point when lab activity slows down. By running the vast majority of our VMs on one system, we can reduce our operating costs with lower cooling and electrical overhead.
A breakdown of some of the VMs always on in the StorageReview Test Lab:
- vCenter 6.0 Appliance: Keeps our VMware hosts in order
- DNS/AD Server
- Eaton IPM: Used to monitor power consumption and control UPSs
- Large File Server VM: Server 2012 R2 w/ Dedupe to hold testing output and other items
- Script Parser: CentOS VM for background tasks and output CSV generation
- Vendor VMs: Staging ground for vendors to keep an eye on their gear during tests
- Veeam Backup VM: Keeping backups in check and routing data to our ExaGrid backup appliance
One of the first upgrades we performed on the 1028U-TNR4T+ was adding a SAS3 RAID card (upgrading from onboard SATA) and installing a 16Gb FC adapter. The RAID card increases the performance potential for on-board SATA/SAS storage as well as offers it into single-volumes for VMware ESXi 6. The FC adapter was brought in to attach the server to the DotHill Ultra48, which provides shared storage amongst all the servers in the lab. With quad 10G ports already included with the server, we didn't need to add additional Ethernet cards for faster iSCSI or vMotion capabilities.
From the rear you can see we've used two PCIe 3.0 slots available in the server, leaving one additional slot open for expansion. It's really hard not to enjoy having 4 10GbE ports on the rear on-board, since that frees up PCIe slots for additional components.
With the eight Seagate 10K.8 1.8TB SAS3 HDDs, we created a large RAID6 pool with 9.8TB of usable space. This volume is used to install ESXi 6.0 on, as well as offer local storage that we use when migrating VMs around the lab.
In the end, with the system brought online, we have a sleek 1U powerhouse, offering 384GB of RAM for plenty of locally hosted VMs that are no problem for the Intel E5-2699 v3 CPUs. With 36-cores total we have nearly 83GHz of CPU headroom in this single server, consuming under 200w with a minimal workload. Supermicro has done a great job providing a flexible platform that's perfect for our lab and will surely find similar use cases for many other datacenters.