Approximating quantiles in very large datasets

Details Approximating quantiles in very large datasets Approximating quantiles in very large datasets

2010-07-07

arxiv.org/abs/1007.1032v1

Statistics

Computation

Scientific paper

Very large datasets are often encountered in climatology, either from a multiplicity of observations over time and space or outputs from deterministic models (sometimes in petabytes= 1 million gigabytes). Loading a large data vector and sorting it, is impossible sometimes due to memory limitations or computing power. We show that a proposed algorithm to approximating the median, "the median of the median" performs poorly. Instead we develop an algorithm to approximate quantiles of very large datasets which works by partitioning the data or use existing partitions (possibly of non-equal size). We show the deterministic precision of this algorithm and how it can be adjusted to get customized precisions.

Affiliated with

Also associated with

No associations

Rating

Approximating quantiles in very large datasets does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Approximating quantiles in very large datasets, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Approximating quantiles in very large datasets will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-344810