Beyond Word N-Grams

Details Beyond Word N-Grams Beyond Word N-Grams

1996-07-13

arxiv.org/abs/cmp-lg/9607016v1

Computer Science

Computation and Language

15 pages, one PostScript figure, uses psfig.sty and fullname.sty. Revised version of a paper in the Proceedings of the Third W

Scientific paper

We describe, analyze, and evaluate experimentally a new probabilistic model for word-sequence prediction in natural language based on prediction suffix trees (PSTs). By using efficient data structures, we extend the notion of PST to unbounded vocabularies. We also show how to use a Bayesian approach based on recursive priors over all possible PSTs to efficiently maintain tree mixtures. These mixtures have provably and practically better performance than almost any single model. We evaluate the model on several corpora. The low perplexity achieved by relatively small PST mixture models suggests that they may be an advantageous alternative, both theoretically and practically, to the widely used n-gram models.

Affiliated with

Also associated with

No associations

Rating

Beyond Word N-Grams does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Beyond Word N-Grams, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Beyond Word N-Grams will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-36840