Computer Science – Databases
Scientific paper
Dec 2005
adsabs.harvard.edu/cgi-bin/nph-data_query?bibcode=2005aspc..347..599l&link_type=abstract
Astronomical Data Analysis Software and Systems XIV ASP Conference Series, Vol. 347, Proceedings of the Conference held 24-27 Oc
Computer Science
Databases
Scientific paper
The UKIRT Wide Field Camera (WFCAM) on Mauna Kea and the VISTA IR mosaic camera at ESO, Paranal, with respectively 4 Rockwell 2k × 2k and 16 Raytheon 2k × 2k IR arrays on 4m-class telescopes, represent an enormous leap in deep IR survey capability. However with an expected data rate of an image of the sky every 5-30s and combined nightly data-rates of typically 1 TB, automated pipeline processing and data management requirements are paramount.
Pipeline processing of IR data is far more technically challenging than for optical data. IR detectors are inherently more unstable, while the sky emission is over 100 times brighter than most objects of interest, and varies in a complex spatial and temporal manner.
The pipelines are designed around a selectable modular scheme, driven by processing recipes for maximum flexibility. Our general philosophy is that all fundamental data products are in multi-extension FITS files with headers describing the data taking protocols in sufficient detail to trigger the appropriate pipeline processing components. All derived information, DQC, photometric and astrometric calibration and processing details are also incorporated into the FITS headers. Generated catalogues are stored in FITS binary tables. The headers provide a basis for ingest into databases for archiving, real time monitoring of survey progress and survey planning.
To reduce the data storage I/O overheads and transport requirements, we intend to use, as much as possible, the lossless Rice tile compression scheme as used transparently, for example, in CFITSIO. For this type of data (32 bit integer) the algorithm typically gives a factor of 3-4 compression
Bunclark P. S.
Evans Dafydd W.
Hodgkin Simon T.
Irwin Mary Jane
Lewis James R.
No associations
LandOfFree
Vista Data Flow System: Pipeline Processing for WFCAM and VISTA does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Vista Data Flow System: Pipeline Processing for WFCAM and VISTA, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Vista Data Flow System: Pipeline Processing for WFCAM and VISTA will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-1035006