Knoche, Holger and Eichelberger, Holger (2018) Using the Raspberry Pi and Docker for Replicable Performance Experiments: Experience Paper. [Paper] In: ACM/SPEC International Conference on Performance Engineering Proceedings of the 2018. , April 09 - 13, 2018, Berlin . Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering. ; pp. 305-316 . DOI 10.1145/3184407.3184431. ICPE '18 .

Preview	Text ICPE2018-Knoche.pdf - Published Version Download (1MB) | Preview
Preview	Slideshow ICPE2018-Knoche-Presentation.pdf - Presentation Download (741kB) | Preview

Abstract

Replicating software performance experiments is difficult. A common obstacle to replication is that recreating the hardware and software environments is often impractical. As researchers usually run their experiments on the hardware and software that happens to be available to them, recreating the experiments would require obtaining identical hardware, which can lead to high costs. Recreating the software environment is also difficult, as software components such as particular library versions might no longer be available. Cheap, standardized hardware components like the Raspberry Pi and portable software containers like the ones provided by Docker are a potential solution to meet the challenge of replicability. In this paper, we report on experiences from replicating performance experiments on Raspberry Pi devices with and without Docker and show that good replication results can be achieved for microbenchmarks such as JMH. Replication of macrobenchmarks like SPECjEnterprise 2010 proves to be much more difficult, as they are strongly affected by (non-standardized) peripherals. Inspired by previous microbenchmarking experiments on the Pi platform, we furthermore report on a systematic analysis of response time fluctuations, and present lessons learned on dos and don'ts for replicable performance experiments.

Actions (login required)

View Item