The Standard Performance Evaluation Corp. (SPEC) has released a major new version of its storage solution benchmark, designed to help IT managers make better purchasing, configuration and optimization decisions.
by Don Capps, SPEC Storage subcommittee chair
The Standard Performance Evaluation Corp. (SPEC) has released a major new version of its storage solution benchmark, designed to help IT managers make better purchasing, configuration and optimization decisions.
The new SPECstorage Solution 2020 benchmark is based on actual applications and real-world scenarios. It includes new workloads for artificial intelligence (AI) and genomics, expanded custom workload capabilities, massively better scaling, and a statistical visualization mechanism for displaying benchmark results. There are also dozens of upgrades that make the benchmark easier to use and more efficient in delivering results.
Workloads within the SPECstorage Solution 2020 benchmark represent applications used within five different markets. Each workload has a set of pass-fail criteria that must be met in order to validate the benchmark results. Examples of criteria include:
The new AI workload represents AI Tensorflow image processing environments, with traces collected from systems running COCO, Resnet50 and CityScape datasets. Metrics are based on the number of simultaneous jobs— defined as a set of subcomponents that encompass one complete workflow — that can be sustained under the defined pass/fail criteria.
The new genomics workload comes from traces of I/O operations used by commercial and research facilities to perform genetic analysis. The data has been sanitized so that it doesn’t contain any of the original genome data. Metrics are based on the number of simultaneous jobs that can be maintained under the workload’s pass-fail criteria.
Three workloads are carried over from the SPEC SFS 2014 SP2 benchmark, released in December 2017:
The new benchmark enables users to customize existing workloads or create new ones to gain insight into storage performance issues.
Customized workloads can accommodate a combination of Unix and Windows load generators for applications that use this combination in their workflows. An example of this combination would be a Windows system that collects sensor data and stores it using SMB while a Unix-based compute farm is analyzing the data using NFS, Lustre, GPFS, or some other file system technology.
A sophisticated synchronization mechanism within the SPECstorage Solution 2020 benchmark keeps all of the geographically distributed load-generating processes in sync at a sub-millisecond resolution.
New internal IPC mechanisms in SPECstorage Solution 2020 significantly reduce the number of TCP ports and DNS lookups, improving scalability, reliability, and start-up and run-time performance. Scaling has been increased from 60,000 load-generating processes in previous versions to more than 4 million processes that can be distributed throughout the world.
A sophisticated synchronization mechanism keeps all of the geographically distributed load-generating processes in sync at a sub-millisecond resolution.
The new scaling efficiency is especially significant for testing cloud-based storage solutions. This is an important development, since among cloud providers, TCP ports are a scarce resource, with most virtual machines able to accommodate only a few thousand at best. The new benchmark never requires more than 1,000 TCP ports. This enables users to scale up to 4 million processes without running into cloud constraints.
The new statistical visualization mechanism within SPECstorage Solution 2020 allows users to extract runtime counters and connect them to a database such as Graphite, Carbon or Grafana for visualization. This provides greater insight into the behavior and operation details of the system under test.
The SPECstorage Solution 2020 benchmark is available for immediate download on the SPEC website for $2,000, with discounts for qualified research and academic institutions. Results from the new benchmark are not comparable to those from past SPEC storage benchmarks.
SPEC corporate members active in the development of the benchmark include Dell, IBM, Intel, iX Systems, NetApp, Pure Storage, and WekaIO. Supporting individual contributors includeUdayan Bapat, Sorin Faibish, and Brian Pawlowski.
###
Don Capps has more than 30 years of experience in performance engineering for storage systems. He is a performance engineer for Netapp; a founder of Iozone.org, a non-profit organization that produces and distributes free software for measuring the performance of computer storage systems; and chair of the Standard Performance Evaluation Corp. (SPEC) Storage subcommittee.
Engage with StorageReview
Newsletter | YouTube | Podcast iTunes/Spotify | Instagram | Twitter | Facebook | RSS Feed
Dell Technologies is advancing its data protection portfolio to enhance cyber resiliency across appliances, software, and as-a-service offerings amid rising…
Since its launch in 2019, the Cray ClusterStor E1000 Storage System has emerged as a pivotal technology in the field…
Quantum Corporation has introduced Quantum GO, a subscription service designed to meet the escalating data demands and cost considerations enterprises…
JetCool has launched an innovative liquid cooling module tailored for NVIDIA's H100 SXM and PCIe GPUs, claiming a significant advancement…
iXsystems has launched the TrueNAS Enterprise H-Series platforms, designed to give organizations ultimate performance. The H10 model is now available,…
Hannover Messe 2024 represents a significant event in the global industrial sector, serving as the world's largest industrial trade fair.…