Computer Science – Networking and Internet Architecture
Scientific paper
2012-01-19
Computer Networks 55, 8 (2011) 1880-1893
Computer Science
Networking and Internet Architecture
Scientific paper
10.1016/j.comnet.2011.01.013
A typical web search engine consists of three principal parts: crawling engine, indexing engine, and searching engine. The present work aims to optimize the performance of the crawling engine. The crawling engine finds new web pages and updates web pages existing in the database of the web search engine. The crawling engine has several robots collecting information from the Internet. We first calculate various performance measures of the system (e.g., probability of arbitrary page loss due to the buffer overflow, probability of starvation of the system, the average time waiting in the buffer). Intuitively, we would like to avoid system starvation and at the same time to minimize the information loss. We formulate the problem as a multi-criteria optimization problem and attributing a weight to each criterion. We solve it in the class of threshold policies. We consider a very general web page arrival process modeled by Batch Marked Markov Arrival Process and a very general service time modeled by Phase-type distribution. The model has been applied to the performance evaluation and optimization of the crawler designed by INRIA Maestro team in the framework of the RIAM INRIA-Canon research project.
Avrachenkov Konstantin
Dudin Alexander
Klimenok Valentina
Nain Philippe
Semenova Olga
No associations
LandOfFree
Optimal Threshold Control by the Robots of Web Search Engines with Obsolescence of Documents does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Optimal Threshold Control by the Robots of Web Search Engines with Obsolescence of Documents, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Optimal Threshold Control by the Robots of Web Search Engines with Obsolescence of Documents will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-254946