A Vector-Scheduling Approach for Running Many-Task Applications in the Cloud
Document Type
Conference Proceeding
Publication Date
1-1-2018
Abstract
The performance variation of cloud resources makes it difficult to run certain scientific applications in the cloud because of their unique synchronization and communication requirements. We propose a decentralized scheduling approach for many-task applications that assigns individual tasks to cloud nodes based on periodic performance measurements of the cloud resources. In this paper, we present a vector-based scheduling algorithm that assigns tasks to nodes based on measuring the compute performance and the queue length of those nodes. Our experiments with a set of tasks in CloudLab show that the application proceeds in three distinct phases: flooding the cloud nodes with tasks, a steady state in which all nodes are busy, and the end game in which the remaining tasks are executed on the fastest nodes. We present heuristics for these three phases and demonstrate with measurements in CloudLab that they result in a reduction of the overall execution time of the many-task application.
Publication Source (Journal or Book title)
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
First Page
3
Last Page
19
Recommended Citation
Peterson, B., Fazlalizadeh, Y., Baumgartner, G., & Wang, Q. (2018). A Vector-Scheduling Approach for Running Many-Task Applications in the Cloud. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10967 LNCS, 3-19. https://doi.org/10.1007/978-3-319-94295-7_1
