Computer Science – Social and Information Networks
Scientific paper
2012-04-16
Computer Science
Social and Information Networks
Presented at Collective Intelligence conference, 2012 (arXiv:1204.2991)
Scientific paper
A common use of crowd sourcing is to obtain labels for a dataset. Several algorithms have been proposed to identify uninformative members of the crowd so that their labels can be disregarded and the cost of paying them avoided. One common motivation of these algorithms is to try and do without any initial set of trusted labeled data. We analyse this class of algorithms as mechanisms in a game-theoretic setting to understand the incentives they create for workers. We find an impossibility result that without any ground truth, and when workers have access to commonly shared 'prejudices' upon which they agree but are not informative of true labels, there is always equilibria where all agents report the prejudice. A small amount amount of gold standard data is found to be sufficient to rule out these equilibria.
Penna Nicolás Della
Reid Mark D.
No associations
LandOfFree
Crowd & Prejudice: An Impossibility Theorem for Crowd Labelling without a Gold Standard does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Crowd & Prejudice: An Impossibility Theorem for Crowd Labelling without a Gold Standard, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Crowd & Prejudice: An Impossibility Theorem for Crowd Labelling without a Gold Standard will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-6807