Efficient and Effective Spam Filtering and Re-ranking for Large Web Datasets

Details Efficient and Effective Spam Filtering and Re-ranking for Large Web Datasets Efficient and Effective Spam Filtering and Re-ranking for Large Web Datasets

2010-04-29

arxiv.org/abs/1004.5168v1

Computer Science

Information Retrieval

Scientific paper

The TREC 2009 web ad hoc and relevance feedback tasks used a new document collection, the ClueWeb09 dataset, which was crawled from the general Web in early 2009. This dataset contains 1 billion web pages, a substantial fraction of which are spam --- pages designed to deceive search engines so as to deliver an unwanted payload. We examine the effect of spam on the results of the TREC 2009 web ad hoc and relevance feedback tasks, which used the ClueWeb09 dataset. We show that a simple content-based classifier with minimal training is efficient enough to rank the "spamminess" of every page in the dataset using a standard personal computer in 48 hours, and effective enough to yield significant and substantive improvements in the fixed-cutoff precision (estP10) as well as rank measures (estR-Precision, StatMAP, MAP) of nearly all submitted runs. Moreover, using a set of "honeypot" queries the labeling of training data may be reduced to an entirely automatic process. The results of classical information retrieval methods are particularly enhanced by filtering --- from among the worst to among the best.

Affiliated with

Also associated with

No associations

Rating

Efficient and Effective Spam Filtering and Re-ranking for Large Web Datasets does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Efficient and Effective Spam Filtering and Re-ranking for Large Web Datasets, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Efficient and Effective Spam Filtering and Re-ranking for Large Web Datasets will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-284352