Reliable, Automatic Transfer and Processing of Large Scale Astronomy Datasets

Details Reliable, Automatic Transfer and Processing of Large Scale Astronomy Datasets Reliable, Automatic Transfer and Processing of Large Scale Astronomy Datasets

Dec 2005

adsabs.harvard.edu/cgi-bin/nph-data_query?bibcode=2005aspc..347..277k&link_type=abstract

Astronomical Data Analysis Software and Systems XIV ASP Conference Series, Vol. 347, Proceedings of the Conference held 24-27 Oc

Astronomy and Astrophysics

Astronomy

Scientific paper

Astronomers are increasingly obtaining larger datasets, particularly in the optical and near-infrared. Unfortunately, the technologies to process large amounts of image data and share the data and the results with collaborators spread around the globe, have not kept pace with the data flow. In the past, this type of software has required significant human involvement to deal with failures. We have designed and implemented a fault-tolerant system that can process large amounts of astronomy images using idle CPUs on desktops, commodity clusters and grid resources. It reliably replicates data and results to collaborating sites and performs on-the-fly optimization to improve throughput. It is highly resilient to failures and can recover automatically from network, storage server, software and hardware failures. To demonstrate the capabilities of this framework, we have successfully processed three terabytes of DPOSS images using idle grid resources spread across three organizations.

Affiliated with

Also associated with

No associations

Rating

Reliable, Automatic Transfer and Processing of Large Scale Astronomy Datasets does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Reliable, Automatic Transfer and Processing of Large Scale Astronomy Datasets, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Reliable, Automatic Transfer and Processing of Large Scale Astronomy Datasets will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-1034612