Computer Science – Performance
Scientific paper
Sep 2009
adsabs.harvard.edu/cgi-bin/nph-data_query?bibcode=2009aspc..411..497s&link_type=abstract
Astronomical Data Analysis Software and Systems XVIII ASP Conference Series, Vol. 411, proceedings of the conference held 2-5 No
Computer Science
Performance
Scientific paper
The NOAO Data Management system (DMS) captures data from eleven NOAO and partner telescopes and transports these data from three mountaintops to replicate them between three data centers both North and South of the equator. Image files are annotated, remediated, ingested, and persisted through interfaces of the NOAO Science Archive. Wide-field optical and infrared images flow out of the archive, through the NOAO High Performance Pipeline creating several new data products that flow back into the archive. Raw, pipeline-reduced, and survey data products, both proprietary and post-proprietary, are made available through the NOAO Portal using VO standards and services.
Each of these several steps requires access to both image data and metadata in the form of image header keywords. Measures of storage efficiency and throughput characterize performance, cost, schedule, and risk in a matrix across all telescopes and all subsystems. Anything that impedes access to data or metadata diminishes throughput, thus slowing schedules, increasing costs, revealing risks, and adversely affecting performance.
The familiar gzip compression algorithm is often used to increase data storage efficiency. However, gzip actually reduces throughput due to initial and recurring overhead of compression and later uncompression. For example, if metadata for an image require remediation, the whole image must be compressed, uncompressed, and compressed again. By contrast, the FITS tile convention using the Rice algorithm achieves about 40% better compression than gzip in just one-third the time. Image headers remain readable such that images often need never be uncompressed at all; metadata can be simply edited in place. Further, a library such as CFITSIO can support tile compression as a native image format. The pixel tiling feature means that for applications such as a cutout service, only the tiles overlapping the desired image section need be uncompressed.
Barg Irene
Seaman Rob
Stobie Elizabeth
No associations
LandOfFree
FITS Tile Compression in the NOAO DMS does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with FITS Tile Compression in the NOAO DMS, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and FITS Tile Compression in the NOAO DMS will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-834683