Towards automatic parameter tuning of stream processing systems

Muhammad Bilal, Marco Canini

Research output: Chapter in Book/Report/Conference proceedingConference contribution

17 Scopus citations

Abstract

Optimizing the performance of big-data streaming applications has become a daunting and time-consuming task: parameters may be tuned from a space of hundreds or even thousands of possible configurations. In this paper, we present a framework for automating parameter tuning for stream-processing systems. Our framework supports standard black-box optimization algorithms as well as a novel gray-box optimization algorithm. We demonstrate the multiple benefits of automated parameter tuning in optimizing three benchmark applications in Apache Storm. Our results show that a hill-climbing algorithm that uses a new heuristic sampling approach based on Latin Hypercube provides the best results. Our gray-box algorithm provides comparable results while being two to five times faster.
Original languageEnglish (US)
Title of host publicationProceedings of the 2017 Symposium on Cloud Computing - SoCC '17
PublisherAssociation for Computing Machinery (ACM)
Pages189-200
Number of pages12
ISBN (Print)9781450350280
DOIs
StatePublished - Sep 27 2017

Fingerprint Dive into the research topics of 'Towards automatic parameter tuning of stream processing systems'. Together they form a unique fingerprint.

Cite this