Column-Oriented Storage Techniques for MapReduce

Details Column-Oriented Storage Techniques for MapReduce Column-Oriented Storage Techniques for MapReduce

2011-05-21

arxiv.org/abs/1105.4252v1

Computer Science

Databases

VLDB2011

Scientific paper

Users of MapReduce often run into performance problems when they scale up their workloads. Many of the problems they encounter can be overcome by applying techniques learned from over three decades of research on parallel DBMSs. However, translating these techniques to a MapReduce implementation such as Hadoop presents unique challenges that can lead to new design choices. This paper describes how column-oriented storage techniques can be incorporated in Hadoop in a way that preserves its popular programming APIs. We show that simply using binary storage formats in Hadoop can provide a 3x performance boost over the naive use of text files. We then introduce a column-oriented storage format that is compatible with the replication and scheduling constraints of Hadoop and show that it can speed up MapReduce jobs on real workloads by an order of magnitude. We also show that dealing with complex column types such as arrays, maps, and nested records, which are common in MapReduce jobs, can incur significant CPU overhead. Finally, we introduce a novel skip list column format and lazy record construction strategy that avoids deserializing unwanted records to provide an additional 1.5x performance boost. Experiments on a real intranet crawl are used to show that our column-oriented storage techniques can improve the performance of the map phase in Hadoop by as much as two orders of magnitude.

Affiliated with

Also associated with

No associations

Rating

Column-Oriented Storage Techniques for MapReduce does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Column-Oriented Storage Techniques for MapReduce, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Column-Oriented Storage Techniques for MapReduce will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-713630