A Robust Text Processing Technique Applied to Lexical Error Recovery

Computer Science – Computation and Language

Scientific paper

Rate now

  [ 0.00 ] – not rated yet Voters 0   Comments 0

Details

Licentiate Thesis, 101 pages

Scientific paper

This thesis addresses automatic lexical error recovery and tokenization of corrupt text input. We propose a technique that can automatically correct misspellings, segmentation errors and real-word errors in a unified framework that uses both a model of language production and a model of the typing behavior, and which makes tokenization part of the recovery process. The typing process is modeled as a noisy channel where Hidden Markov Models are used to model the channel characteristics. Weak statistical language models are used to predict what sentences are likely to be transmitted through the channel. These components are held together in the Token Passing framework which provides the desired tight coupling between orthographic pattern matching and linguistic expectation. The system, CTR (Connected Text Recognition), has been tested on two corpora derived from two different applications, a natural language dialogue system and a transcription typing scenario. Experiments show that CTR can automatically correct a considerable portion of the errors in the test sets without introducing too much noise. The segmentation error correction rate is virtually faultless.

No associations

LandOfFree

Say what you really think

Search LandOfFree.com for scientists and scientific papers. Rate them and share your experience with other people.

Rating

A Robust Text Processing Technique Applied to Lexical Error Recovery does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with A Robust Text Processing Technique Applied to Lexical Error Recovery, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and A Robust Text Processing Technique Applied to Lexical Error Recovery will most certainly appreciate the feedback.

Rate now

     

Profile ID: LFWR-SCP-O-89176

  Search
All data on this website is collected from public sources. Our data reflects the most accurate information available at the time of publication.