
Benchmarking Procedure
The benchmarking procedure is standardized and well defined. It is monitored by the SAP benchmark council made up of representatives of SAP and technology partners involved in benchmarking.
Originally introduced to strengthen quality assurance, the SAP Standard Application Benchmarks can also be used to test and verify scalability, concurrency, power efficiency, and multi-user behavior of system software components, RDBMS, and business applications.
While the majority of benchmarks are run online, the SAP APO, SAP BW, HR (Payroll), PS, Retail, and ISU/CCS Benchmarks are run in the background, and the TRBK Benchmark has both an online section and a batch section.
Over the years, specific sets of configurations have become the de facto standard, independent of the hardware vendor, operating system, or database version. To simplify certification, these configuration examples are used as a default. Any deviation from this default needs to be stated explicitly.
However, as hardware and technology architectures evolve, the defaults are also subject to changes. The default configurations and major changes are described below.
Codepage
To come in line with the overall move of SAP applications to Unicode, all SAP Standard Application Benchmarks use Unicode from June 1, 2010 onwards. Prior to that, some of the benchmarks used the Latin-1 code page, and if Unicode was used in these benchmarks, a special mention was made on the benchmark certificate. For other benchmarks, e.g. the Enterprise Portal benchmarks, there was no need to mention Unicode explicitly on the certificate as no other option was possible.
Note that for the ERP benchmarks, in particular the SD benchmark, Unicode was explicitly mentioned on the certificate for a transition period of one year, i.e. throughout 2009.
32-bit/64-bit software version
Early SAP software releases supported only 32-bit versions of operating systems and databases. With improved hardware capabilities 64-bit versions of OS and DB software are supported and thus the SAP software can run on 64-bit versions of OS and DB as well. SAP software on 64-bit Unix versions has been available since 1999, and became default in 2001. 64-bit Windows and Linux versions became available in 2002.
In the benchmark environment whether 32-bit or 64-bit versions are used, strongly depends on the SAP release and the processor type. As a general rule, you could say that as of mySAP ERP 2005 (ECC 6.0) and SAP NetWeaver 2004s the default used is 64-bit. Before that time, roughly since 1999 there was a coexistence of benchmarks run on either version, predominantly depending on processor type. In this interim period there were processors that used only 32-bit, only 64-bit, and some that supported both. As a consequence, the benchmark certificates detailed the use of 32-bit and 64-bit, respectively, if the processor supported both versions. When a benchmark was run on a processor that supported only one of the versions, it was not specifically mentioned, as there was no option for the submitter.
Hardware configuration
The benchmark certificates describe the total server configuration of a benchmark setup. This includes all active servers, and for each active server the total number of processors, cores, threads as well as the total amount of memory, no matter whether this was used during the actual benchmark or not.
Also, in the benchmark certificates, "segmentation" refers to the kind of partitioning, virtualization, or other technology that may have been used to run more than one operating system image on a physical server.
The details of the actual benchmark configuration can be obtained from SAP or the respective technology partner.
Number of processors
With the development of new processor architectures using multi-core and multi-threading technology, the n-way SMP description was no longer sufficient to describe the processing units. Since August 2005 the certificate states the total number of processors / cores / threads. These numbers describe the maximum of the hardware no matter whether all processors / cores / threads are being used or not. The actual configuration during the benchmark may be different.
Note: The terms CPU, processors, etc. might be used differently by the individual vendor.
Virtualized platforms
When a benchmark was run on a virtualized environment the certificate can be extended to include the virtualization technology. The extended certificate states the virtualization technology, the entitlement or the number of assigned processing units to the virtualized environment, and the CPU utilization within the virtualized environment.
Note: The server configuration on the benchmark certificate describes the complete hardware, whether or not it is used. For example, the actual configuration used by the virtualized environment during the benchmark may be different from the complete hardware.
Identical Servers
In cases where vendors use one certification for identical servers these servers are detailed on the benchmark certificate. In addition see the statements on the respective hardware vendor’s Web page:
- 2007017, 2007021,
- 2007030, 2007031,
- 2007041, 2007043,
- 2007075, 2008022,
- 2007026, 2007051,
- 2007059, 2008021,
- 2008040, 2008042,
- 2008058, 2008062,
- 2008075, 2009026,
- 2009038, 2009046,
- 2021001, 2021002
- 2022020, 2022021
Partner Trademarks/ Copyrights
Some products mentioned on the SAP Standard Application Benchmarks pages and on SAP Benchmark Certifications are trademarks or registered trademarks of other vendors.
To learn about copyrights/trademarks for IBM, Microsoft, Open Group, and Oracle, please view the copyrights and copyright agent page.
AMD, the AMD Arrow logo, AMD Opteron, and combinations thereof are trademarks of Advanced Micro Devices Inc. Other names are for informational purposes only and may be trademarks of their respective owners.
PRIMERGY and PRIMEPOWER are trademarks of Fujitsu Siemens Computers.
Most HP trademarks are on file with the U.S. Patent and Trademark Office, typically in the name of Hewlett-Packard Development Company, L.P., a Texas Limited Partnership. HP does not use or require the use of trademark legends or acknowledgements unless agreed to by both parties involved in the development of a specific business or legal contract.
Intel® Xeon™, Pentium® II Xeon™, Pentium® III Xeon™, Itanium®, and Itanium® 2 are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
Sun, Sun Microsystems, the Sun logo, iForce, Netra, N1, Solaris, Sun Fire, Sun Ray, SunSpectrum, Sun StorEdge, SunTone, The Network is the Computer, all trademarks and logos that contain Sun, Solaris, or Java, and certain other trademarks and logos are trademarks or registered trademarks of Sun Microsystems Inc. in the United States and other countries. See also the copyrights and copyright agent page.
All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc. in the United States and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems Inc.
This trademark information is in addition, and not in lieu of, any and all information contained in the SAP privacy statement
Obtain more information
According to the full disclosure policy, you can obtain more information about a particular benchmark from SAP and the respective technology partner.