Computer Science – Computation and Language
Scientific paper
2010-08-31
Computer Science
Computation and Language
Scientific paper
Lexical co-occurrence is an important cue for detecting word associations. We present a theoretical framework for discovering statistically significant lexical co-occurrences from a given corpus. In contrast with the prevalent practice of giving weightage to unigram frequencies, we focus only on the documents containing both the terms (of a candidate bigram). We detect biases in span distributions of associated words, while being agnostic to variations in global unigram frequencies. Our framework has the fidelity to distinguish different classes of lexical co-occurrences, based on strengths of the document and corpuslevel cues of co-occurrence in the data. We perform extensive experiments on benchmark data sets to study the performance of various co-occurrence measures that are currently known in literature. We find that a relatively obscure measure called Ochiai, and a newly introduced measure CSA capture the notion of lexical co-occurrence best, followed next by LLR, Dice, and TTest, while another popular measure, PMI, suprisingly, performs poorly in the context of lexical co-occurrence.
Chaudhari Dipak
Damani Om P.
Laxman Srivatsan
No associations
LandOfFree
Lexical Co-occurrence, Statistical Significance, and Word Association does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Lexical Co-occurrence, Statistical Significance, and Word Association, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Lexical Co-occurrence, Statistical Significance, and Word Association will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-180306