Bayesian Stratified Sampling to Assess Corpus Utility

Details Bayesian Stratified Sampling to Assess Corpus Utility Bayesian Stratified Sampling to Assess Corpus Utility

1998-06-19

arxiv.org/abs/cmp-lg/9806012v1

Computer Science

Computation and Language

8 pages, 5 figures. To appear in Proceedings of WVLC-6

Scientific paper

This paper describes a method for asking statistical questions about a large text corpus. We exemplify the method by addressing the question, "What percentage of Federal Register documents are real documents, of possible interest to a text researcher or analyst?" We estimate an answer to this question by evaluating 200 documents selected from a corpus of 45,820 Federal Register documents. Stratified sampling is used to reduce the sampling uncertainty of the estimate from over 3100 documents to fewer than 1000. The stratification is based on observed characteristics of real documents, while the sampling procedure incorporates a Bayesian version of Neyman allocation. A possible application of the method is to establish baseline statistics used to estimate recall rates for information retrieval systems.

Affiliated with

Also associated with

No associations

Rating

Bayesian Stratified Sampling to Assess Corpus Utility does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Bayesian Stratified Sampling to Assess Corpus Utility, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Bayesian Stratified Sampling to Assess Corpus Utility will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-22779