Weight decay induced phase transitions in multilayer neural networks

Details Weight decay induced phase transitions in multilayer neural networks Weight decay induced phase transitions in multilayer neural networks

1999-01-19

arxiv.org/abs/cond-mat/9901179v1

Physics

Condensed Matter

Disordered Systems and Neural Networks

7 pages, 7 figures

Scientific paper

10.1088/0305-4470/32/27/301

We investigate layered neural networks with differentiable activation function and student vectors without normalization constraint by means of equilibrium statistical physics. We consider the learning of perfectly realizable rules and find that the length of student vectors becomes infinite, unless a proper weight decay term is added to the energy. Then, the system undergoes a first order phase transition between states with very long student vectors and states where the lengths are comparable to those of the teacher vectors. Additionally in both configurations there is a phase transition between a specialized and an unspecialized phase. An anti-specialized phase with long student vectors exists in networks with a small number of hidden units.

Affiliated with

Also associated with

No associations

Rating

Weight decay induced phase transitions in multilayer neural networks does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.

If you have personal experience with Weight decay induced phase transitions in multilayer neural networks, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Weight decay induced phase transitions in multilayer neural networks will most certainly appreciate the feedback.

Rate now

Comments { 0 }

Profile ID: LFWR-SCP-O-668677