Dynamic algorithm selection for the logic of tasks in IoT stream processing systems

Various Internet of Things and Industry 4.0 use cases such as city-wide monitoring, Smart Grid control, or machine control, require low-latency distributed processing of continuous data streams. This fact has boosted research on making Stream Processing Frameworks (SPFs) IoT-ready, meaning that their cloud and IoT service management mechanisms (e.g., task placement, load balancing, algorithm selection) need to consider new requirements derived from IoT-specific characteristics, i.e., ultra low latency due to physical interactions. Although various extensions have appeared to optimize such SPF-provided mechanisms, they still lack the modules, data models, and algorithms to properly handle algorithm selection in IoT deployments. The algorithm selection problem refers to selecting dynamically which internal logic a deployed streaming task should use in case of various alternatives. To the best of our knowledge, this work is the first solution that adds this capability to SPFs. Our solution is based on i) architectural extensions of typical SPF middleware, ii) a new schema for characterizing algorithmic performance in the targeted context, and iii) a streaming-specific optimization problem formulation. We implemented our solution as an extension to Apache Storm and demonstrate how it can reduce stream processing latency by up to a factor of 2.9 in the tested scenarios.