This work supports users by reacting to changing runtime conditions through automatically adjusting configurations of distributed stream processing systems such as Apache Flink and Spark Streaming.
Distributed Stream Processing systems have become an increasingly critical part of modern big data processing platforms. It is here where high throughput rates, low end-to-end latencies, and the ability to continue operating in the presence of partial failures are essential for supporting time-sensitive decision making. However, the complexity with which these systems are composed has given rise to a multitude of configuration options which are little understood even by experts and greatly impact upon the Quality of Service they are expected to provide. The heterogeneous nature of stream processing environments makes using a one-size-fits-all approach to configuration highly inefficient whereas finding individually optimized setups are difficult and time consuming. This coupled with the fact that streaming workloads vary over time making static configurations obsolete almost as soon as the job is started.
In this work we develop adaptive approaches to multi-configuration optimization for Distributed Stream Processing jobs executing in environments where workloads are expected to change over time. We make use of profiling, runtime modeling, and Time Series Forecasting to provide solutions where both performance and availability configuration sets are automatically adjusted at runtime to ensure alignment with Quality of Service requirements. Importantly, we not only consider what configurations should be changed, but also when these changes should take place. We provide prototypes and together with modern Distributed Stream Processing systems such as Apache Flink and Spark Streaming demonstrate their usefulness experimentally.
If you have any questions or are interested in collaborating with us on this topic, please get in touch with Morgan!
This work has been supported through grants by the German Ministry for Education and Research (BMBF) as BIFOLD (funding mark 01IS18025A) and WaterGridSense 4.0 (funding mark 02WIK1475D).