Extending the SDSS Batch Query System to the National Virtual Observatory Grid

Details Extending the SDSS Batch Query System to the National Virtual Observatory Grid Extending the SDSS Batch Query System to the National Virtual Observatory Grid

2004-03-12

arxiv.org/abs/cs/0403017v1

Computer Science

Databases

original available at http://research.microsoft.com/research/pubs/view.aspx?tr_id=714

Scientific paper

The Sloan Digital Sky Survey science database is approaching 2TB. While the vast majority of queries normally execute in seconds or minutes, this interactive execution time can be disproportionately increased by a small fraction of queries that take hours or days to run; either because they require non-index scans of the largest tables or because they request very large result sets. In response to this, we added a multi-queue job submission and tracking system. The transfer of very large result sets from queries over the network is another serious problem. Statistics suggested that much of this data transfer is unnecessary; users would prefer to store results locally in order to allow further cross matching and filtering. To allow local analysis, we implemented a system that gives users their own personal database (MyDB) at the portal site. Users may transfer data to their MyDB, and then perform further analysis before extracting it to their own machine. We intend to extend the MyDB and asynchronous query ideas to multiple NVO nodes. This implies development, in a distributed manner, of several features, which have been demonstrated for a single node in the SDSS Batch Query System (CasJobs). The generalization of asynchronous queries necessitates some form of MyDB storage as well as workflow tracking services on each node and coordination strategies among nodes.

Affiliated with

Also associated with

No associations

Rating

Extending the SDSS Batch Query System to the National Virtual Observatory Grid does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Extending the SDSS Batch Query System to the National Virtual Observatory Grid, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Extending the SDSS Batch Query System to the National Virtual Observatory Grid will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-161353