Computer Science – Distributed – Parallel – and Cluster Computing
Scientific paper
2010-01-21
Computer Science
Distributed, Parallel, and Cluster Computing
13 pages, 7 figures
Scientific paper
With a goal of supporting the timely and cost-effective analysis of Terabyte datasets on commodity components, we present and evaluate StoreTorrent, a simple distributed filesystem with integrated fault tolerance for efficient handling of small data records. Our contributions include an application-OS pipelining technique and metadata structure to increase small write and read performance by a factor of 1-10, and the use of peer-to-peer communication of replica-location indexes to avoid transferring data during parallel analysis even in a degraded state. We evaluated StoreTorrent, PVFS, and Gluster filesystems using 70 storage nodes and 560 parallel clients on an 8-core/node Ethernet cluster with directly attached SATA disks. StoreTorrent performed parallel small writes at an aggregate rate of 1.69 GB/s, and supported reads over the network at 8.47 GB/s. We ported a parallel analysis task and demonstrate it achieved parallel reads at the full aggregate speed of the storage node local filesystems.
No associations
LandOfFree
Performance and Fault Tolerance in the StoreTorrent Parallel Filesystem does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Performance and Fault Tolerance in the StoreTorrent Parallel Filesystem, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Performance and Fault Tolerance in the StoreTorrent Parallel Filesystem will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-249339