One-Pass, One-Hash n-Gram Statistics Estimation

Details One-Pass, One-Hash n-Gram Statistics Estimation One-Pass, One-Hash n-Gram Statistics Estimation

2006-10-03

arxiv.org/abs/cs/0610010v3

Computer Science

Databases

The bibliography was not properly updated in the previous submission

Scientific paper

In multimedia, text or bioinformatics databases, applications query sequences of n consecutive symbols called n-grams. Estimating the number of distinct n-grams is a view-size estimation problem. While view sizes can be estimated by sampling under statistical assumptions, we desire an unassuming algorithm with universally valid accuracy bounds. Most related work has focused on repeatedly hashing the data, which is prohibitive for large data sources. We prove that a one-pass one-hash algorithm is sufficient for accurate estimates if the hashing is sufficiently independent. To reduce costs further, we investigate recursive random hashing algorithms and show that they are sufficiently independent in practice. We compare our running times with exact counts using suffix arrays and show that, while we use hardly any storage, we are an order of magnitude faster. The approach further is extended to a one-pass/one-hash computation of n-gram entropy and iceberg counts. The experiments use a large collection of English text from the Gutenberg Project as well as synthetic data.

Affiliated with

Also associated with

No associations

Rating

One-Pass, One-Hash n-Gram Statistics Estimation does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with One-Pass, One-Hash n-Gram Statistics Estimation, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and One-Pass, One-Hash n-Gram Statistics Estimation will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-588947