When majority voting fails: Comparing quality assurance methods for noisy human computation environment

Details When majority voting fails: Comparing quality assurance methods for noisy human computation environment When majority voting fails: Comparing quality assurance methods for noisy human computation environment

2012-04-16

arxiv.org/abs/1204.3516v1

Computer Science

Social and Information Networks

Presented at Collective Intelligence conference, 2012 (arXiv:1204.2991)

Scientific paper

Quality assurance remains a key topic in human computation research. Prior work indicates that majority voting is effective for low difficulty tasks, but has limitations for harder tasks. This paper explores two methods of addressing this problem: tournament selection and elimination selection, which exploit 2-, 3- and 4-way comparisons between different answers to human computation tasks. Our experimental results and statistical analyses show that both methods produce the correct answer in noisy human computation environment more often than majority voting. Furthermore, we find that the use of 4-way comparisons can significantly reduce the cost of quality assurance relative to the use of 2-way comparisons.

Affiliated with

Also associated with

No associations

Rating

When majority voting fails: Comparing quality assurance methods for noisy human computation environment does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with When majority voting fails: Comparing quality assurance methods for noisy human computation environment, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and When majority voting fails: Comparing quality assurance methods for noisy human computation environment will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-6831