Computer Science – Distributed – Parallel – and Cluster Computing
Scientific paper
2006-02-07
Computer Science
Distributed, Parallel, and Cluster Computing
4 pages, 5 figures. Accepted by the Computing for High Energy Physics Conference. Mumbai, Indai. February 2006
Scientific paper
Results from and progress on the development of a Data Intensive and Network Aware (DIANA) Scheduling engine, primarily for data intensive sciences such as physics analysis, are described. Scientific analysis tasks can involve thousands of computing, data handling, and network resources and the size of the input and output files and the amount of overall storage space allocated to a user necessarily can have significant bearing on the scheduling of data intensive applications. If the input or output files must be retrieved from a remote location, then the time required transferring the files must also be taken into consideration when scheduling compute resources for the given application. The central problem in this study is the coordinated management of computation and data at multiple locations and not simply data movement. However, this can be a very costly operation and efficient scheduling can be a challenge if compute and data resources are mapped without network cost. We have implemented an adaptive algorithm within the DIANA Scheduler which takes into account data location and size, network performance and computation capability to make efficient global scheduling decisions. DIANA is a performance-aware as well as an economy-guided Meta Scheduler. It iteratively allocates each job to the site that is likely to produce the best performance as well as optimizing the global queue for any remaining pending jobs. Therefore it is equally suitable whether a single job is being submitted or bulk scheduling is being performed. Results suggest that considerable performance improvements are to be gained by adopting the DIANA scheduling approach.
Ali Arshad
Anjum Ashiq
McClatchey Richard
Willers Ian
No associations
LandOfFree
Bulk Scheduling with DIANA Scheduler does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Bulk Scheduling with DIANA Scheduler, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Bulk Scheduling with DIANA Scheduler will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-63161