Home > EMC, NAS, NetApp, SAN > The Importance of Storage Performance Benchmarks

The Importance of Storage Performance Benchmarks

As one who scans the daily flood of storage news every day, I’ve started to notice an uptick in the number of articles and press releases over the past year highlighting various vendors who have “blown away” a benchmark score of some sort and claim ultimate superiority in the storage world. 2 weeks later, another vendor is trumping that they’ve beaten the score that was just posted 2 weeks prior. With the numbers we’re seeing touted, I’m sure 1 bazillion IOPS must only be right around the corner.

Most vendors who utilize benchmarks tend to be storage startups, looking to get some publicity for themselves, not that there’s anything wrong with that. You gotta get your name out there somehow. For the longest time, the dominant player in the storage world, EMC, refused to participate in benchmarks saying they were not realistic of real-world performance. I don’t disagree with that, in many cases benchmarks are not indicative of real-world performance. Nevertheless, now even EMC has jumped into the fray. Perhaps they decided that not participating costs more in negative press than it does good.

What does it all mean for you? Here are a couple things to consider:

  1. Most benchmark tests are not indicative of real-world results. If you want to use a benchmark stat to get a better sense of what the max system limits are, that’s fine. But, don’t forget what your requirements truly are and measure each system against that. In most cases, customers are using a storage array for mixed workloads from a variety of business apps for a variety of use cases (database, email, file, and VMware, etc).  These different applications all have different I/O patterns.  The benchmark tests don’t simulate workloads that are related to this “real-world” mixed I/O pattern.   Benchmarks are heavily tilted in favor of niche use cases with very specific workloads. I’m sure there are niche use cases out there where the benchmarks do matter, but for 95% of storage buyers, they don’t matter. The bottom line is be sure the system has enough bandwidth and spindles to handle your real MB/sec and IOPS requirements. Designing that properly will be much more beneficial to you than getting an array that recently did 1 million IOPS in a benchmark test.
  1. Every vendor will reference a benchmark that works in their favor. Every vendor pretty much seems to be able to pull a benchmark stat out of their hat that favors their systems above all others.

Here’s an example I saw last year when working with a customer who evaluated 3 different vendors (IBM, NetApp, and EMC), and how I helped the customer get clear on what was real. In this case, both non-EMC vendors were referencing SPC numbers that showed how badly an EMC CX3-40 performed relative to their platforms. A couple alarms went off for me immediately:

  1. The CX3-40 was a generation old relative to the other platforms. The CX4 was the current platform on the market (now replaced by VNX). In other words, not an apples-to-apples comparison.
  2. At the time the CX3-40 existed, EMC did not participate in SAN benchmarks for its mid-range or high-end arrays.

I took a look at the V7000 SPC-1 benchmark and found some interesting conclusions.  Here is a chart that shows how the V7000 performed on the benchmark, and it shows other competitors as well:


The V7000 scored 56,500.   Interesting to note, since the box only supported 120 drives at the time, they had to utilize the SVC code in it to add on a DS5020 box, which allowed them to add more drives (200 total) to the configuration.  They put 80 15K RPM drives in the DS5020, higher speed drives the V7000 didn’t support natively at the time.    What’s important to note about the CX3-40 results seen in the SPC1 results is that this was a CX3-40 that NetApp purchased and ran a test on, then submitted the results to without EMC’s permission.  I don’t care what your vendor affiliation is, that’s not a fair fight. EMC had no input into how the array was configured and tuned.    Even though the array could hold 240 drives, it was only configured it for 155.    The CX3-40 scored 25,000.   Let’s make a realistic assumption that if EMC had configured the array and tuned it for the benchmark as other vendors did, then it could have done at least 25% better.   This would give it a score of 31,000.    The CX3-40 was a predecessor to the CX4-240 and they both hold 240 drives.   Performance and spec limits pretty much doubled across the board from CX3 to CX4, because EMC implemented a 64-bit architecture with the CX4’s release in 2008. So, again let’s make a realistic assumption and take the 31,000 result of the CX3-40 and double it to create a theoretical score for the CX4-240 of 62,000.

If I look at other arrays that are comparable to the CX4-240 in the results list, such as the DS5300 or FAS 3000 series, this theoretical score is right in the ballpark of the other arrays.     I would hope most would agree that this shows all the arrays in-scope were within striking distance of each other.   What exactly do these numbers mean relative to your business….not much. You can’t design a system for your business needs using these numbers. When most customers are analyzing their performance requirements, they have figures for IOPS, throughput, and latency that they need to meet to ensure good business application performance, not a theoretical benchmark score target.

Benchmarks can certainly be interesting, and I admit sometimes I think it’s cool to see a system that did X number of GB’s per second of throughput or X million IOPS, but my recommendation is don’t get to spun up on them in your search for a storage platform.  Every vendor has a benchmark that makes them look the best.  Instead, use your own metrics or work with a trusted business partner who can help you gather the data specific to your environment and evaluate each technology against how well it meets your business needs.

Advertisements
Categories: EMC, NAS, NetApp, SAN
  1. November 28, 2011 at 5:22 AM

    Hi Dan,

    Great read, I’m deffinately adding you to my list of blogs to read.

    I’ve just written up a piece on the CX3-40 and though you might find it equally as interesting.

    http://ausstorageguy.wordpress.com/2011/11/26/dragging-up-ancient-history-how-netapp-fooled-everyone-fas3040-v-cx3-40/

    Keep up the good work.

  2. November 28, 2011 at 11:07 AM

    Thanks for the comments, and great work on that article. That is an incredible level of detail that I never knew (PCI-E vs. PCI-X, EMC config had PS dollars in it but the config did not, and so on). Look forward to more posts!

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: