Computer Science – Computation and Language
Scientific paper
2002-05-10
Natural Language Engineering 9 (2), pp. 127--149, 2003
Computer Science
Computation and Language
22 pages. To appear in Natural Language Engineering
Scientific paper
10.1017/S1351324902002954
Given the lack of word delimiters in written Japanese, word segmentation is generally considered a crucial first step in processing Japanese texts. Typical Japanese segmentation algorithms rely either on a lexicon and syntactic analysis or on pre-segmented data; but these are labor-intensive, and the lexico-syntactic techniques are vulnerable to the unknown word problem. In contrast, we introduce a novel, more robust statistical method utilizing unsegmented training data. Despite its simplicity, the algorithm yields performance on long kanji sequences comparable to and sometimes surpassing that of state-of-the-art morphological analyzers over a variety of error metrics. The algorithm also outperforms another mostly-unsupervised statistical algorithm previously proposed for Chinese. Additionally, we present a two-level annotation scheme for Japanese to incorporate multiple segmentation granularities, and introduce two novel evaluation metrics, both based on the notion of a compatible bracket, that can account for multiple granularities simultaneously.
Ando Rie Kubota
Lee Lillian
No associations
LandOfFree
Mostly-Unsupervised Statistical Segmentation of Japanese Kanji Sequences does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Mostly-Unsupervised Statistical Segmentation of Japanese Kanji Sequences, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Mostly-Unsupervised Statistical Segmentation of Japanese Kanji Sequences will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-382469