Statistical Dynamics of On-line Independent Component Analysis

Details Statistical Dynamics of On-line Independent Component Analysis Statistical Dynamics of On-line Independent Component Analysis

2003-09-24

arxiv.org/abs/cond-mat/0309554v1

Physics

Condensed Matter

Disordered Systems and Neural Networks

18 pages, 13 figures, to appear in Journal of Machine Learning Research special issue on ICA

Scientific paper

The learning dynamics of on-line independent component analysis is analysed in the limit of large data dimension. We study a simple Hebbian learning algorithm that can be used to separate out a small number of non-Gaussian components from a high-dimensional data set. The de-mixing matrix parameters are confined to a Stiefel manifold of tall, orthogonal matrices and we introduce a natural gradient variant of the algorithm which is appropriate to learning on this manifold. For large input dimension the parameter trajectory of both algorithms passes through a sequence of unstable fixed points, each described by a diffusion process in a polynomial potential. Choosing the learning rate too large increases the escape time from each of these fixed points, effectively trapping the learning in a sub-optimal state. In order to avoid these trapping states a very low learning rate must be chosen during the learning transient, resulting in learning time-scales of $O(N^2)$ or $O(N^3)$ iterations where $N$ is the data dimension. Escape from each sub-optimal state results in a sequence of symmetry breaking events as the algorithm learns each source in turn. This is in marked contrast to the learning dynamics displayed by related on-line learning algorithms for multilayer neural networks and principal component analysis. Although the natural gradient variant of the algorithm has nice asymptotic convergence properties, it has an equivalent transient dynamics to the standard Hebbian algorithm.

Affiliated with

Also associated with

No associations

Rating

Statistical Dynamics of On-line Independent Component Analysis does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Statistical Dynamics of On-line Independent Component Analysis, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Statistical Dynamics of On-line Independent Component Analysis will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-509745