Approximating Document Frequency with Term Count Values

Details Approximating Document Frequency with Term Count Values Approximating Document Frequency with Term Count Values

2008-07-23

arxiv.org/abs/0807.3755v1

Computer Science

Information Retrieval

11 pages, 6 figures, 4 tables

Scientific paper

For bounded datasets such as the TREC Web Track (WT10g) the computation of term frequency (TF) and inverse document frequency (IDF) is not difficult. However, when the corpus is the entire web, direct IDF calculation is impossible and values must instead be estimated. Most available datasets provide values for term count (TC) meaning the number of times a certain term occurs in the entire corpus. Intuitively this value is different from document frequency (DF), the number of documents (e.g., web pages) a certain term occurs in. We conduct a comparison study between TC and DF values within the Web as Corpus (WaC). We found a very strong correlation with Spearman's rho >0.8 (p<0.005) which makes us confident in claiming that for such recently created corpora the TC and DF values can be used interchangeably to compute IDF values. These results are useful for the generation of accurate lexical signatures based on the TF-IDF scheme.

Affiliated with

Also associated with

No associations

Rating

Approximating Document Frequency with Term Count Values does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Approximating Document Frequency with Term Count Values, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Approximating Document Frequency with Term Count Values will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-675660