Challenges and Experience in Prototyping a Multi-Modal Stream Analytic and Monitoring Application on System S

In this paper, we describe the challenges of prototyping a reference application on System S, a distributed stream processing middle-ware under development at IBM Research. With a large number of stream PEs (Processing Elements) implementing various stream analytic algorithms, running on a large-scale, distributed cluster of nodes, and collaboratively digesting several multi-modal source streams with vastly divergent rates, prototyping a reference application on System S faces many challenges. Specifically, we focus on three critical challenges of, and our experience in, prototyping DAC (Disaster Assistance Claim monitoring), a reference application dealing with multi-modal stream analytic and monitoring. The three challenges are: (1) How do we generate correlated, multi-modal source streams for DAC? (2) How do we design and implement a comprehensive stream application, like DAC, from many divergent stream analytic PEs? (3) How do we deploy DAC in light of source streams with extremely different rates? We report our experience in addressing these challenges, including modeling a disaster claim processing center to generate correlated source streams, constructing the PE flow graph, utilizing programming supports from System S, adopting parallelism, and exploiting resource-adaptive computation.

By: Kun-Lung Wu, Philip S. Yu, Bugra Gedik, Kirsten W. Hildrum, Charu C. Aggarwal, Eric Bouillet, Wei Fan, David A. George, Xiaohui Gu, Gang Luo, Haixun Wang

Published in: RC24199 in 2007


This Research Report is available. This report has been submitted for publication outside of IBM and will probably be copyrighted if accepted for publication. It has been issued as a Research Report for early dissemination of its contents. In view of the transfer of copyright to the outside publisher, its distribution outside of IBM prior to publication should be limited to peer communications and specific requests. After outside publication, requests should be filled only by reprints or legally obtained copies of the article (e.g., payment of royalties). I have read and understand this notice and am a member of the scientific community outside or inside of IBM seeking a single copy only.


Questions about this service can be mailed to .