Computer Science – Distributed – Parallel – and Cluster Computing
Scientific paper
2005-02-17
Computer Science
Distributed, Parallel, and Cluster Computing
Scientific paper
The Sloan Digital Sky Survey (SDSS) science database describes over 140 million objects and is over 1.5 TB in size. The SDSS Catalog Archive Server (CAS) provides several levels of query interface to the SDSS data via the SkyServer website. Most queries execute in seconds or minutes. However, some queries can take hours or days, either because they require non-index scans of the largest tables, or because they request very large result sets, or because they represent very complex aggregations of the data. These "monster queries" not only take a long time, they also affect response times for everyone else - one or more of them can clog the entire system. To ameliorate this problem, we developed a multi-server multi-queue batch job submission and tracking system for the CAS called CasJobs. The transfer of very large result sets from queries over the network is another serious problem. Statistics suggested that much of this data transfer is unnecessary; users would prefer to store results locally in order to allow further joins and filtering. To allow local analysis, a system was developed that gives users their own personal databases (MyDB) at the server side. Users may transfer data to their MyDB, and then perform further analysis before extracting it to their own machine. MyDB tables also provide a convenient way to share results of queries with collaborators without downloading them. CasJobs is built using SOAP XML Web services and has been in operation since May 2004.
Gray Jim
Li Nolan
Nieto-Santisteban Maria
O'Mullane William
Szalay Alex
No associations
LandOfFree
Batch is back: CasJobs, serving multi-TB data on the Web does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Batch is back: CasJobs, serving multi-TB data on the Web, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Batch is back: CasJobs, serving multi-TB data on the Web will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-97230