Drop the SPC smoke screen

Sent to you by Vachak via Google Reader: 

via Storage Soup by Dave Raffo on 9/12/08


Heweltt-Packard has fired the latest shot at EMC in the battle over performance benchmarks. HP this week posted records for megabytes per second and price performance in SPC-2 performance benchmark testing of its XP24000 enterprise SAN array, and immediately called out EMC for its refusal to submit its products to the Storage Performance Council (SPC) for benchmarking.

According to a blog by Craig Simpson, competitive strategist for HP StorageWorks:

EMC, we’re once again throwing down the gauntlet.  Today the XP24000 put up the highest SPC-2 benchmark result in the world.  The top spot for such demanding workloads as video streaming goes to the XP.  Once again, your DMX is a no show.  And once again we challenge you, this time to put up an SPC-2 number.  Every other major storage vendor is now posting SPC results.  Every other major storage vendor is now starting to give customers standard, open, audited performance results to show what they’ve got.  You remain the only vendor keeping your product performance behind a smoke screen of mysterious numbers and internal testing.  We challenge you join us in the world of openness and let customers quit guessing at how low the DMX’s performance really is!

Interestingly, the XP24000 isn’t HP’s own system. It is sourced from Hitachi Japan, and sold by Hitachi Data Systems and Sun as well as HP. And HP’s SPC-1 mark for random I/O operations (SPC-2 is for sequential data movement) was recently surpassed by 3PAR’s InServ Storage System.

But from HP’s standpoint, this isn’t about HDS, Sun, or 3PAR. It’s about going after EMC, which remains resolute in its refusal to take part in SPC testing.

“An oversimplified performance test that doesn’t accurately predict real-world performance is of little value to customers,” an EMC spokesman said in response to HP’s latest challenge.

Until now, the benchmarking skirmish was mainly between NetApp and EMC. It’s been going on for years, but last February NetApp took it to another level last February when it published benchmarks for EMC’s Clariion  CX3-40 that showed it performing worse than NetApp’s FAS3040.

EMC blogger Chuck Hollis then came up with his own “standardized measure” for storage capacity efficiency last month. He pulled HP into the fray by comparing EMC CX4 against NetApp FAS and HP EVA series. (Spoiler alert: EMC came out on top).

And if EMC’s results shock you, then I’m sure you’re equally stunned to learn that HP and NetApp took exception to EMC’s numbers.

“Capacity utilization is important, but there’s no third-party body out there that measures cap utilization,” Simpson said. “We felt Chuck’s position was very skewed. We would love to see them agree to have an independent third-party to pick up the challenge.”

Does anybody besides vendors care about these things? I asked Babu Kudaravalli, senior director of operations for Business Technology Services at Port Washington, NY-based National Medical Health Card, if benchmarks were a factor in his buying storage systems. He did say he found SPC-2 more relevant to him than many enterprise shops because he runs large queries that entail sequential data. But Kudaravelli bought his two XP24000s last year, long before the latest SPC-2 numbers were released, so he saw the numbers more as vindication than as a buying guide.

“We pay attention to it, but don’t go purely based on SPC numbers,” he said. “Sometimes benchmarks are not relevant, but I was thrilled when I saw the SPC-2 number. When I saw the results, I said I already bought a winner.”

No comments: