Data Processing Factory for the Sloan Digital Sky Survey

Details Data Processing Factory for the Sloan Digital Sky Survey Data Processing Factory for the Sloan Digital Sky Survey

Dec 2002

adsabs.harvard.edu/cgi-bin/nph-data_query?bibcode=2002spie.4836..339s&link_type=abstract

Survey and Other Telescope Technologies and Discoveries. Edited by Tyson, J. Anthony; Wolff, Sidney. Proceedings of the SPIE,

Other

11

Scientific paper

The Sloan Digital Sky Survey (SDSS) data handling presents two challenges: large data volume and timely production of spectroscopic plates from imaging data. A data processing factory, using technologies both old and new, handles this flow. Distribution to end users is via disk farms, to serve corrected images and calibrated spectra, and a database, to efficiently process catalog queries. For distribution of modest amounts of data from Apache Point Observatory to Fermilab, scripts use rsync to update files, while larger data transfers are accomplished by shipping magnetic tapes commercially. All data processing pipelines are wrapped in scripts to address consecutive phases: preparation, submission, checking, and quality control. We constructed the factory by chaining these pipelines together while using an operational database to hold processed imaging catalogs. The science database catalogs all imaging and spectroscopic object, with pointers to the various external files associated with them. Diverse computing systems address particular processing phases. UNIX computers handle tape reading and writing, as well as calibration steps that require access to a large amount of data with relatively modest computational demands. Commodity CPUs process steps that require access to a limited amount of data with more demanding computations requirements. Disk servers optimized for cost per Gbyte serve terabytes of processed data, while servers optimized for disk read speed run SQLServer software to process queries on the catalogs. This factory produced data for the SDSS Early Data Release in June 2001, and it is currently producing Data Release One, scheduled for January 2003.

Affiliated with

Also associated with

No associations

Rating

Data Processing Factory for the Sloan Digital Sky Survey does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Data Processing Factory for the Sloan Digital Sky Survey, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Data Processing Factory for the Sloan Digital Sky Survey will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-1310594