Statistics – Computation
Scientific paper
Jan 2011
adsabs.harvard.edu/cgi-bin/nph-data_query?bibcode=2011aas...21734412w&link_type=abstract
American Astronomical Society, AAS Meeting #217, #344.12; Bulletin of the American Astronomical Society, Vol. 43, 2011
Statistics
Computation
Scientific paper
In the coming decade, astronomical surveys of the sky will generate tens of terabytes of images and detect hundreds of millions of sources every night. The study of these sources will involve computational challenges such as anomaly detection, classification, and moving object tracking. Since such studies require the highest quality data, methods such as image coaddition, i.e., registration, stacking, and mosaicing, will be critical to scientific investigation. With a requirement that these images be analyzed on a nightly basis to identify moving sources, e.g., asteroids, or transient objects, e.g., supernovae, these datastreams present many computational challenges. Given the quantity of data involved, the computational load of these problems can only be addressed by distributing the workload over a large number of nodes. However, the high data throughput demanded by these applications may present scalability challenges for certain storage architectures. One scalable data-processing method that has emerged in recent years is MapReduce, and in this paper we focus on its popular open-source implementation called Hadoop. In the Hadoop framework, the data is partitioned among storage attached directly to worker nodes, and the processing workload is scheduled in parallel on the nodes that contain the required input data. A further motivation for using Hadoop is that it allows us to exploit cloud computing resources, i.e., platforms where Hadoop is offered as a service. We report on our experience implementing a scalable image-processing pipeline for the SDSS imaging database using Hadoop. This multi-terabyte imaging dataset provides a good testbed for algorithm development since its scope and structure approximate future surveys. First, we describe MapReduce and how we adapted image coaddition to the MapReduce framework. Then we describe a number of optimizations to our basic approach and report experimental results compring their performance. This work is funded by the NSF and by NASA.
Balazinska Magdalena
Bu Yan Yan
Connolly Amy
Gardner James
Howe Bill
No associations
LandOfFree
Astronomy In The Cloud: Using Mapreduce For Image Coaddition does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Astronomy In The Cloud: Using Mapreduce For Image Coaddition, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Astronomy In The Cloud: Using Mapreduce For Image Coaddition will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-1403147