Computer Science – Distributed – Parallel – and Cluster Computing
Scientific paper
2001-02-20
Computer Science
Distributed, Parallel, and Cluster Computing
7 pages + title page
Scientific paper
Many scientific applications are I/O intensive and generate or access large data sets, spanning hundreds or thousands of "files." Management, storage, efficient access, and analysis of this data present an extremely challenging task. We have developed a software system, called Scientific Data Manager (SDM), that uses a combination of parallel file I/O and database support for high-performance scientific data management. SDM provides a high-level API to the user and internally, uses a parallel file system to store real data and a database to store application-related metadata. In this paper, we describe how we designed and implemented SDM to support irregular applications. SDM can efficiently handle the reading and writing of data in an irregular mesh as well as the distribution of index values. We describe the SDM user interface and how we implemented it to achieve high performance. SDM makes extensive use of MPI-IO's noncontiguous collective I/O functions. SDM also uses the concept of a history file to optimize the cost of the index distribution using the metadata stored in the database. We present performance results with two irregular applications, a CFD code called FUN3D and a Rayleigh-Taylor instability code, on the SGI Origin2000 at Argonne National Laboratory.
Choudhary Alok
Freitag Lori
Kaushik Dinesh
No Jaechun
Thakur Rajeev
No associations
LandOfFree
A Scientific Data Management System for Irregular Applications does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with A Scientific Data Management System for Irregular Applications, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and A Scientific Data Management System for Irregular Applications will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-520099