Computer Science – Computation and Language
Scientific paper
2004-11-20
Computer Science
Computation and Language
9 pages 3 figures 2 tables
Scientific paper
Chinese text segmentation is a well-known and difficult problem. On one side, there is not a simple notion of "word" in Chinese language making really hard to implement rule-based systems to segment written texts, thus lexicons and statistical information are usually employed to achieve such a task. On the other side, any piece of Chinese text usually includes segments present neither in the lexicons nor in the training data. Even worse, such unseen sequences can be segmented into a number of totally unrelated words making later processing phases difficult. For instance, using a lexicon-based system the sequence ???(Baluozuo, Barroso, current president-designate of the European Commission) can be segmented into ?(ba, to hope, to wish) and ??(luozuo, an undefined word) changing completely the meaning of the sentence. A new and extremely simple algorithm specially suited to work over short Chinese documents is introduced. This new algorithm performs text "self-segmentation" producing results comparable to those achieved by native speakers without using either lexicons or any statistical information beyond the obtained from the input text. Furthermore, it is really robust for finding new "words", especially proper nouns, and it is well suited to build lexicons from scratch. Some preliminary results are provided in addition to examples of its employment.
No associations
LandOfFree
Building Chinese Lexicons from Scratch by Unsupervised Short Document Self-Segmentation does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Building Chinese Lexicons from Scratch by Unsupervised Short Document Self-Segmentation, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Building Chinese Lexicons from Scratch by Unsupervised Short Document Self-Segmentation will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-169846