Designing for Peta-Scale in the LSST Database

Details Designing for Peta-Scale in the LSST Database Designing for Peta-Scale in the LSST Database

Oct 2007

adsabs.harvard.edu/cgi-bin/nph-data_query?bibcode=2007aspc..376....3k&link_type=abstract

Astronomical Data Analysis Software and Systems XVI ASP Conference Series, Vol. 376, proceedings of the conference held 15-18 Oc

Computer Science

Databases

1

Scientific paper

The Large Synoptic Survey Telescope (LSST), a proposed ground-based 8.4 m telescope with a 10 deg^2 field of view, will generate 15 TB of raw images every observing night. When calibration and processed data are added, the image archive, catalogs, and meta-data will grow 15 PB yr^{-1} on average. The LSST Data Management System (DMS) must capture, process, store, index, replicate, and provide open access to this data. Alerts must be triggered within 30 s of data acquisition. To do this in real-time at these data volumes will require advances in data management, database, and file system techniques.
This paper describes the design of the LSST DMS and emphasizes features for peta-scale data. The LSST DMS will employ a combination of distributed database and file systems, with schema, partitioning, and indexing oriented for parallel operations. Image files are stored in a distributed file system with references to, and meta-data from, each file stored in the databases. The schema design supports pipeline processing, rapid ingest, and efficient query. Vertical partitioning reduces disk input/output requirements, horizontal partitioning allows parallel data access using arrays of servers and disks. Indexing is extensive, utilizing both conventional RAM-resident indexes and column-narrow, row-deep tag tables/covering indices that are extracted from tables that contain many more attributes.
The DMS Data Access Framework is encapsulated in a middleware framework to provide a uniform service interface to all framework capabilities. This framework will provide the automated work-flow, replication, and data analysis capabilities necessary to make data processing and data quality analysis feasible at this scale.

Affiliated with

Also associated with

No associations

Rating

Designing for Peta-Scale in the LSST Database does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Designing for Peta-Scale in the LSST Database, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Designing for Peta-Scale in the LSST Database will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-955610