Computer Science – Computation and Language
Scientific paper
2011-05-31
Computer Science
Computation and Language
30 pages, 5 figures
Scientific paper
In natural speech, the speaker does not pause between words, yet a human listener somehow perceives this continuous stream of phonemes as a series of distinct words. The detection of boundaries between spoken words is an instance of a general capability of the human neocortex to remember and to recognize recurring sequences. This paper describes a computer algorithm that is designed to solve the problem of locating word boundaries in blocks of English text from which the spaces have been removed. This problem avoids the complexities of speech processing but requires similar capabilities for detecting recurring sequences. The algorithm relies entirely on statistical relationships between letters in the input stream to infer the locations of word boundaries. A Viterbi trellis is used to simultaneously evaluate a set of hypothetical segmentations of a block of adjacent words. This technique improves accuracy but incurs a small latency between the arrival of letters in the input stream and the sending of words to the output stream. The source code for a C++ version of this algorithm is presented in an appendix.
No associations
LandOfFree
A statistical learning algorithm for word segmentation does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with A statistical learning algorithm for word segmentation, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and A statistical learning algorithm for word segmentation will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-337668