From the abstract
Our results demonstrate that measurement bias is significant and commonplace in computer system evaluation. By significant we mean that measurement bias can lead to a performance analysis that either over-states an effect or even yields an incorrect conclusion. By commonplace we mean that measurement bias occurs in all architectures that we tried (Pentium 4, Core 2, and m5 O3CPU), both compilers that we tried (gcc and Intel’s C compiler), and most of the SPEC CPU2006 C programs. Thus, we cannot ignore measurement bias. Nevertheless, in a literature survey of 133 recent papers from ASPLOS, PACT, PLDI, and CGO, we determined that none of the papers with experimental results adequately consider measurement bias.
The Emery Berger talk at Strange Loop 2019, “Performance Matters” is the continuation of this research and well worth watching.
More like this (3)
In Tenable.io, we are heavy users of Datadog custom metrics. Millions of metrics are sent through...In Tenable.io, we are heavy users of Datadog custom metrics. Millions of metrics are sent through Dogstatsd, providing deep insights into the complex platform. As the platform grew, we found that a significant number of metrics sent by legacy apps were obsolete. We tried to hunt down these obsoleted metrics in the c...
It happens once in a while that a significant archive of a colonial businessmen gets discovered...It happens once in a while that a significant archive of a colonial businessmen gets discovered in an attic. The caretakers of a manor house – the Huseby Bruk – in the Småland region of South Sweden discovered boxes full of papers and objects, lying silently in an upper story dark hall. These were the India papers o...
Credit: University of Michigan Edsger Dijkstra’s 1988 paper "On the cruelty of really teaching computer science"...Credit: University of Michigan Edsger Dijkstra’s 1988 paper "On the cruelty of really teaching computer science" (in plain text form here) is one of the most well-cited papers on computer science (CS) education. It’s also wrong. A growing body of recent research explores the very topic that Dijkstra tried to warn us...