With SAP HANA Benchmark comparisons, sometimes “the devil is in the details”

People often use the expression “the devil is in the details” when something looks good on the surface but when you look closer, you see hidden tricks or “a catch” that make it less attractive. Usually it’s a warning to pay close attention to details that might cause an undesirable result. Until now, I would never have thought that applies to SAP HANA benchmark testing.

A few weeks ago, I published a blog “One Simple Tip for Faster SAP HANA Performance” where I demonstrated how SUSE Linux Enterprise Server 12 had 10% better throughput and a 70% shorter runtime than a comparable server running RHEL 7.4. Well imagine my surprise when I learned that Red Hat had just published a blog with a new benchmark that achieved 53% better throughput and 3% shorter runtime than SLES 12. Wow! Maybe the SUSE engineers need to get back to work on our performance. Or maybe not.

First, I looked at SAP’s BWH benchmark configurations on the website to confirm that this really is a fair comparison.

Both tests were run using the Version 2 benchmark on SAP HANA 1.0 with 1.3M initial records. While the test with RHEL was run on a Dell server and the test with SLES was on a Fujitsu server, both had the same underlying hardware configuration:

• 4 processors / 112 cores / 224 threads 2.5 GHz Intel Xeon Platinum 8180
• 64 KB L1 cache and 1024 KB L2 cache per core, 38.3 MB L3 cache per processor
• 1536 GB main memory

It looks fair so far.

Then I wondered if Red Hat had seen similar results with other benchmarks so I used the CSV download feature on the SAP page (circled in green in the upper right corner of the diagram). As I filtered the results to show common configurations, I noticed an entry in the “Comments” column of this new benchmark that reads “Materializing of the intermediate result of the query was enabled (phase 2 and phase 3)”. One of our SAP specialists explained what this means using simple words that even a marketing manager like me could understand.  Essentially, query materialization stores intermediate query results in the database … sort of like a cache. This results in faster performance because SAP HANA runs differently at the application level, completely independent of the underlying hardware or OS. If you compare the latest benchmark results with materialization on RHEL (Cert # 2018023) with the previous RHEL results on a comparable system without materialization (Cert # 2018019), RHEL 7.4 outperforms itself with 41% better throughput and 75% faster runtime. This is clearly not a valid comparison.

The Red Hat blog goes on to cite another comparison where RHEL supposedly outperforms SLES by a smaller margin, but again it’s those annoying details lurking under the surface. The Dell server is running with RHEL on four Intel Xeon Platinum 8180M processors (Certification # 2018028) but the HPE server is running SLES on four Xeon Platinum 8180 processors (Certification # 2018025). The Xeon 8180M is inherently faster than the 8180, as reported in this CPU World article.

Therefore, I stand by what I said in the previous blog. If you’re looking at benchmarks to choose the fastest SAP HANA platform, pay attention to those configuration details to be sure the comparison is valid. Regardless of the server you choose, stick with SUSE Linux Enterprise Server for SAP Applications for superior performance, reliability and ease of deployment.

Oh, and I tweet from time to time at @MichaelDTabron.