Characterizing, Constructing and Managing Resource Usage Profiles of System S Applications: Challenges and Experience

We describe the challenges of and our experience in characterizing, constructing and managing the usage profiles of System S applications. A System S application deployed at runtime is a directed graph with software processing elements (PEs) as vertices and data streams as edges connecting the PEs. The resource usage of each PE is a critical input to the runtime scheduler for proper resource allocation. We represent the resource usage of PEs in terms of resource functions (RFs) that are used by the System S scheduler, with one RF per resource per PE. The first challenge is that building good RFs that can accurately predict the resource usage of a PE can be difficult because the PEs perform arbitrary computations. A second set of challenges arises in managing the RFs and data so that we can apply them for PEs that are re-run and/or reused by the same or different applications or users. We report our experience in overcoming these challenges. Specifically, we present an empirical characterization of PE RFs from several real streaming applications running in a System S testbed. This justifies our simple, yet effective, models of resource usage that build on the data-flow nature of the underlying application. We show that simple piecewise linear models are generally effective in practice, even for complex PEs. To illustrate our methodology, we evaluate and analyze the performance of several real System S applications as a function of the quality of our resource profile models. To obtain these resource profiles, the system automatically learns the models from the raw metrics data collected from running PEs. We describe our approach to managing the metrics and RF models, which allows us to construct generalizable RFs and eliminates or reduces the learning time for new PEs by intelligently storing and reusing the metrics data.

By: Kirsten W. Hildrum; Deepak Rajan; Sujay Parekh; Joel L. Wolf; Kun-Lung Wu

Published in: RC24667 in 2008


This Research Report is available. This report has been submitted for publication outside of IBM and will probably be copyrighted if accepted for publication. It has been issued as a Research Report for early dissemination of its contents. In view of the transfer of copyright to the outside publisher, its distribution outside of IBM prior to publication should be limited to peer communications and specific requests. After outside publication, requests should be filled only by reprints or legally obtained copies of the article (e.g., payment of royalties). I have read and understand this notice and am a member of the scientific community outside or inside of IBM seeking a single copy only.


Questions about this service can be mailed to .