Computer Science – Learning
Scientific paper
2010-10-20
Proceeding of the 8th International Conference on Information Technology : New Generations, April 11-13, 2011, Las Vegas, Neva
Computer Science
Learning
Scientific paper
This paper considers the clustering problem for large data sets. We propose an approach based on distributed optimization. The clustering problem is formulated as an optimization problem of maximizing the classification gain. We show that the optimization problem can be reformulated and decomposed into small-scale sub optimization problems by using the Dantzig-Wolfe decomposition method. Generally speaking, the Dantzig-Wolfe method can only be used for convex optimization problems, where the duality gaps are zero. Even though, the considered optimization problem in this paper is non-convex, we prove that the duality gap goes to zero, as the problem size goes to infinity. Therefore, the Dantzig-Wolfe method can be applied here. In the proposed approach, the clustering problem is iteratively solved by a group of computers coordinated by one center processor, where each computer solves one independent small-scale sub optimization problem during each iteration, and only a small amount of data communication is needed between the computers and center processor. Numerical results show that the proposed approach is effective and efficient.
No associations
LandOfFree
Large-Scale Clustering Based on Data Compression does not yet have a rating. At this time, there are no reviews or comments for this scientific paper.
If you have personal experience with Large-Scale Clustering Based on Data Compression, we encourage you to share that experience with our LandOfFree.com community. Your opinion is very important and Large-Scale Clustering Based on Data Compression will most certainly appreciate the feedback.
Profile ID: LFWR-SCP-O-717344