Programming abstractions for data intensive computing on clouds and grids
Document Type
Conference Proceeding
Publication Date
10-13-2009
Abstract
MapReduce has emerged as an important dataparallel programming model for data-intensive computing - for Clouds and Grids. However most if not all implementations of MapReduce are coupled to a specific infrastructure. SAGA is a high-level programming interface which provides the ability to create distributed applications in an infrastructure independent way. In this paper, we show how MapReduce has been implemented using SAGA and demonstrate its interoperability across different distributed platforms - Grids, Cloud-like infrastructure and Clouds. We discuss the advantages of programmatically developing MapReduce using SAGA, by demonstrating that the SAGA-based implementation is infrastructure independent whilst still providing control over the deployment, distribution and runtime decomposition. The ability to control the distribution and placement of the computation units (workers) is critical in order to implement the ability to move computational work to the data. This is required to keep data network transfer low and in the case of commercial Clouds the monetary cost of computing the solution low. Using data-sets of size up to 10GB, and up to 10 workers, we provide detailed performance analysis of the SAGA-MapReduce implementation, and show how controlling the distribution of computation and the payload per worker helps enhance performance. © 2009 IEEE.
Publication Source (Journal or Book title)
2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, CCGRID 2009
First Page
478
Last Page
483
Recommended Citation
Miceli, C., Miceli, M., Jha, S., Kaiser, H., & Merzky, A. (2009). Programming abstractions for data intensive computing on clouds and grids. 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, CCGRID 2009, 478-483. https://doi.org/10.1109/CCGRID.2009.87