A pseudo-parallel Python environment for database curation

Details A pseudo-parallel Python environment for database curation A pseudo-parallel Python environment for database curation

2007-11-13

arxiv.org/abs/0711.2042v1

Astronomy and Astrophysics

Astrophysics

4 pages, 2 figures, ADASS XVII conference proceeding. ASP conference series

Scientific paper

One of the major challenges providing large databases like the WFCAM Science Archive (WSA) is to minimize ingest times for pixel/image metadata and catalogue data. In this article we describe how the pipeline processed data are ingested into the database as the first stage in building a release database which will be succeeded by advanced processing (source merging, seaming, detection quality flagging etc.). To accomplish the ingestion procedure as fast as possible we use a mixed Python/C++ environment and run the required tasks in a simple parallel modus operandi where the data are split into daily chunks and then processed on different computers. The created data files can be ingested into the database immediately as they are available. This flexible way of handling the data allows the most usage of the available CPUs as the comparison with sequential processing shows.

Affiliated with

Also associated with

No associations

Rating

A pseudo-parallel Python environment for database curation does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with A pseudo-parallel Python environment for database curation, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and A pseudo-parallel Python environment for database curation will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-271700