TY - GEN
T1 - Towards automatic parameter tuning of stream processing systems
AU - Bilal, Muhammad
AU - Canini, Marco
N1 - KAUST Repository Item: Exported on 2020-12-16
Acknowledgements: Muhammad Bilal was supported by a fellowship from the Erasmus Mundus Joint Doctorate in Distributed Computing (EMJD-DC) program funded by the European Commission (EACEA) (FPA 2012-0030).
PY - 2017/9/27
Y1 - 2017/9/27
N2 - Optimizing the performance of big-data streaming applications has become a daunting and time-consuming task: parameters may be tuned from a space of hundreds or even thousands of possible configurations. In this paper, we present a framework for automating parameter tuning for stream-processing systems. Our framework supports standard black-box optimization algorithms as well as a novel gray-box optimization algorithm. We demonstrate the multiple benefits of automated parameter tuning in optimizing three benchmark applications in Apache Storm. Our results show that a hill-climbing algorithm that uses a new heuristic sampling approach based on Latin Hypercube provides the best results. Our gray-box algorithm provides comparable results while being two to five times faster.
AB - Optimizing the performance of big-data streaming applications has become a daunting and time-consuming task: parameters may be tuned from a space of hundreds or even thousands of possible configurations. In this paper, we present a framework for automating parameter tuning for stream-processing systems. Our framework supports standard black-box optimization algorithms as well as a novel gray-box optimization algorithm. We demonstrate the multiple benefits of automated parameter tuning in optimizing three benchmark applications in Apache Storm. Our results show that a hill-climbing algorithm that uses a new heuristic sampling approach based on Latin Hypercube provides the best results. Our gray-box algorithm provides comparable results while being two to five times faster.
UR - http://hdl.handle.net/10754/626125
UR - https://dl.acm.org/citation.cfm?doid=3127479.3127492
UR - http://www.scopus.com/inward/record.url?scp=85032434738&partnerID=8YFLogxK
U2 - 10.1145/3127479.3127492
DO - 10.1145/3127479.3127492
M3 - Conference contribution
AN - SCOPUS:85032434738
SN - 9781450350280
SP - 189
EP - 200
BT - Proceedings of the 2017 Symposium on Cloud Computing - SoCC '17
PB - Association for Computing Machinery (ACM)
ER -