Accelerating Large-scale Data Exploration through Data Diffusion

Details Accelerating Large-scale Data Exploration through Data Diffusion Accelerating Large-scale Data Exploration through Data Diffusion

2008-08-26

arxiv.org/abs/0808.3546v1

Computer Science

Distributed, Parallel, and Cluster Computing

IEEE/ACM International Workshop on Data-Aware Distributed Computing 2008

Scientific paper

Data-intensive applications often require exploratory analysis of large datasets. If analysis is performed on distributed resources, data locality can be crucial to high throughput and performance. We propose a "data diffusion" approach that acquires compute and storage resources dynamically, replicates data in response to demand, and schedules computations close to data. As demand increases, more resources are acquired, thus allowing faster response to subsequent requests that refer to the same data; when demand drops, resources are released. This approach can provide the benefits of dedicated hardware without the associated high costs, depending on workload and resource characteristics. The approach is reminiscent of cooperative caching, web-caching, and peer-to-peer storage systems, but addresses different application demands. Other data-aware scheduling approaches assume dedicated resources, which can be expensive and/or inefficient if load varies significantly. To explore the feasibility of the data diffusion approach, we have extended the Falkon resource provisioning and task scheduling system to support data caching and data-aware scheduling. Performance results from both micro-benchmarks and a large scale astronomy application demonstrate that our approach improves performance relative to alternative approaches, as well as provides improved scalability as aggregated I/O bandwidth scales linearly with the number of data cache nodes.

Affiliated with

Also associated with

No associations

Rating

Accelerating Large-scale Data Exploration through Data Diffusion does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Accelerating Large-scale Data Exploration through Data Diffusion, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Accelerating Large-scale Data Exploration through Data Diffusion will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-281981