Aggregation for Regression Learning

Details Aggregation for Regression Learning Aggregation for Regression Learning

2004-10-07

arxiv.org/abs/math/0410214v1

Mathematics

Statistics Theory

Scientific paper

This paper studies statistical aggregation procedures in regression setting. A motivating factor is the existence of many different methods of estimation, leading to possibly competing estimators. We consider here three different types of aggregation: model selection (MS) aggregation, convex (C) aggregation and linear (L) aggregation. The objective of (MS) is to select the optimal single estimator from the list; that of (C) is to select the optimal convex combination of the given estimators; and that of (L) is to select the optimal linear combination of the given estimators. We are interested in evaluating the rates of convergence of the excess risks of the estimators obtained by these procedures. Our approach is motivated by recent minimax results in Nemirovski (2000) and Tsybakov (2003). There exist competing aggregation procedures achieving optimal convergence separately for each one of (MS), (C) and (L) cases. Since the bounds in these results are not directly comparable with each other, we suggest an alternative solution. We prove that all the three optimal bounds can be nearly achieved via a single "universal" aggregation procedure. We propose such a procedure which consists in mixing of the initial estimators with the weights obtained by penalized least squares. Two different penalities are considered: one of them is related to hard thresholding techniques, the second one is a data dependent $L_1$-type penalty. Consequently, our method can be endorsed by both the proponents of model selection and the advocates of model averaging.

Affiliated with

Also associated with

No associations

Rating

Aggregation for Regression Learning does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Aggregation for Regression Learning, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Aggregation for Regression Learning will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-216793