Computer Science – Distributed – Parallel – and Cluster Computing
Scientific paper
2008-08-26
Computer Science
Distributed, Parallel, and Cluster Computing
16 pages, 15 figures
Scientific paper
Data intensive applications often involve the analysis of large datasets that require large amounts of compute and storage resources. While dedicated compute and/or storage farms offer good task/data throughput, they suffer low resource utilization problem under varying workloads conditions. If we instead move such data to distributed computing resources, then we incur expensive data transfer cost. In this paper, we propose a data diffusion approach that combines dynamic resource provisioning, on-demand data replication and caching, and data locality-aware scheduling to achieve improved resource efficiency under varying workloads. We define an abstract "data diffusion model" that takes into consideration the workload characteristics, data accessing cost, application throughput and resource utilization; we validate the model using a real-world large-scale astronomy application. Our results show that data diffusion can increase the performance index by as much as 34X, and improve application response time by over 506X, while achieving near-optimal throughputs and execution times.
Foster Ian
Raicu Ioan
Szalay Alex
Zhao Yong
No associations
LandOfFree
Data Diffusion: Dynamic Resource Provision and Data-Aware Scheduling for Data Intensive Applications does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Data Diffusion: Dynamic Resource Provision and Data-Aware Scheduling for Data Intensive Applications, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Data Diffusion: Dynamic Resource Provision and Data-Aware Scheduling for Data Intensive Applications will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-281767