Date: Fri, 27 Mar 92 16:23:20 CST From: jlarson@csrd.uiuc.edu Subject: BenchMARKETing Here is the article on benchmarketing that I made available at the NCSA Academic Affiliates meeting on March 27, 1992. Benchmarketing: The Art of Selling Inferior Goods John L. Larson CSRD, University of Illinois at Urbana-Champaign Benchmarking is that part of performance evaluation that deals with the measurement and analysis of computer performance using various kinds of test programs. Benchmarking becomes "benchmarketing" when the normally objective evaluation process becomes flawed ( incomplete, irrelevant, or invalid ). A naive or intentionally flawed evaluation may be used to convince the believing consumer of the availability of elixors for all ailments[1]. The presentation materials at the end of this paper were produced in December 1990 and have been used by the author in several presentations on various performance topics. These previously unpublished materials have been referenced in the literature[2], and are being published here for wider distribution. An electronic version of the materials is available from the author by email at jlarson@csrd.uiuc.edu. The 10 techniques described in the presentation are benchmarketing techniques witnessed personally by the author in various conference presentations and journal articles. Each technique describes the "logical" process of a benchmarking experiment leading to the same conclusion - "machine A is faster than machine B." A corollary to each conclusion carries the logical process further to a more general conclusion. It is hoped that the reader will see the obvious flaw in the general conclusion, and that the original conclusion is just as flawed. With the currently widespread and growing use of these incorrect performance evaluation techniques has come an increased concern that some controls are necessary to protect the general consumer from those persons who practice this new unscientific discipline. The major conclusion from a recent workshop on benchmarking[3] was that guidelines need to be established and distributed to referees of technical papers for the correct and unbiased measurement and reporting of performance information. It is hoped that the techniques illustrated here will increase your performance literacy, and expose to the benchmarketeers that the consumers are wise to their tricks. References [1] David H. Bailey, "How to fool the masses when giving performance results on parallel computers," Supercomputer, Vol. 8, No. 5, September 1991, pp. 4-7. [2] Jack Worlton, "Toward a taxonomy of performance metrics," Parallel Computing, Vol. 17, No. 10&11, December 1991, pp. 1073-1092. [3] Workshop on the International Coordination of Supercomputer Benchmarking Activities, John Larson and Dave Schneider, Moderators, Supercomputing '91, Albuquerque, NM, November 20, 1991. Benchmarketing: The Art of Selling Inferior Goods Technique 1 - Compare all of A with only part of B * let A consist of many, slow parts * let B consist of a few, fast parts * show that all of the parts of A together are faster than just one part of B * Conclusion: A is faster than B * Corollary: Comparing all of B to one part of A is absurd Technique 2 - Compare A at its best with B at its worst * let A have its data in registers * let B have its data on punched cards in another country * measure the speed of execution * Conclusion: A is faster than B * Corollary: A has enough registers for all real programs Technique 3 - Measure an interesting, but irrelevant attribute * let A have a vector/scalar rate ratio of 100/1 * let B have a vector/scalar rate ratio of 50/5 * measure exection rate for 99.9% vector code * Conclusion: A is faster than B * Corollary: A high ratio indicates fast vector, not slow scalar Technique 4 - Compare a designed feature of A with an accidental feature of B * let A have hardware for performing inverse hyperbolic trigonometric functions in polar coordinates * let B be lucky enough to have software to do the same thing * compare the execution speed * Conclusion: A is faster than B * Corollary: A's advantage is generalizable to all other situations Technique 5 - Compare using an incorrectly applied metric * run 2 benchmark examples: BMK1 and BMK2 * let A have rate 100 on BMK1, and rate 1 on BMK2 * let B have rate 10 on BMK1, and rate 10 on BMK2 * compute ( arithmetic ) average rate: A = 50.5, B = 10.0 * Conclusion: A is 5 times faster than B * Corollary: Performance at the high end is more important than performance at the low end ( complement to Amdahl's Law ) Technique 6 - Compare an optimized A with a default B * expend much effort on A to choose the best algorithm, the best coding practices, the best compiler options, and execute on the largest configuration of hardware * execute B as is, on whatever is convenient * measure the performance * Conclusion: A is faster than B * Corollary: A is easy to use Technique 7 - Quote unobtainable capabilities * let A have n functional units, each of which can produce one result per clock period * calculate n * ( 1/CP ) as the asymtotic peak rate for A, disregarding A's inability to continuously supply operands to all the functional units at this rate * measure actual performance of B * Conclusion: A is faster than B * Corollary: Overall speed is determined by the fastest part ( complement to Amdahl's Law ) Technique 8 - Keep nothing constant * use A to compute matrix multiplication using an assembly language library routine * use B to compute recurrences in FORTRAN * measure the performance * Conclusion: A is faster than B * Corollary: Apples and oranges are both fruits Technique 9 - Compare what A will be with what B is now * announce the availability of A in 3 years * run benchmarks on B * compare execution speeds * Conclusion: A is faster than B * Corollary: All of tomorrow's problems were solved yesterday Technique 10 - Compare A with B's predecessor * run benchmarks on A * recall performance tables from benchmark articles on the Illiac I * compare the performance * Conclusion: A is faster than the HAL-9000 * Corollary: All machines at the University of Illinois are slow