Analysis of correlation based dimension reduction methods

Yong Joon Shin; Cheong Hee Park

International Journal of Applied Mathematics and Computer Science (2011)

  • Volume: 21, Issue: 3, page 549-558
  • ISSN: 1641-876X

Abstract

top
Dimension reduction is an important topic in data mining and machine learning. Especially dimension reduction combined with feature fusion is an effective preprocessing step when the data are described by multiple feature sets. Canonical Correlation Analysis (CCA) and Discriminative Canonical Correlation Analysis (DCCA) are feature fusion methods based on correlation. However, they are different in that DCCA is a supervised method utilizing class label information, while CCA is an unsupervised method. It has been shown that the classification performance of DCCA is superior to that of CCA due to the discriminative power using class label information. On the other hand, Linear Discriminant Analysis (LDA) is a supervised dimension reduction method and it is known as a special case of CCA. In this paper, we analyze the relationship between DCCA and LDA, showing that the projective directions by DCCA are equal to the ones obtained from LDA with respect to an orthogonal transformation. Using the relation with LDA, we propose a new method that can enhance the performance of DCCA. The experimental results show that the proposed method exhibits better classification performance than the original DCCA.

How to cite

top

Yong Joon Shin, and Cheong Hee Park. "Analysis of correlation based dimension reduction methods." International Journal of Applied Mathematics and Computer Science 21.3 (2011): 549-558. <http://eudml.org/doc/208069>.

@article{YongJoonShin2011,
abstract = {Dimension reduction is an important topic in data mining and machine learning. Especially dimension reduction combined with feature fusion is an effective preprocessing step when the data are described by multiple feature sets. Canonical Correlation Analysis (CCA) and Discriminative Canonical Correlation Analysis (DCCA) are feature fusion methods based on correlation. However, they are different in that DCCA is a supervised method utilizing class label information, while CCA is an unsupervised method. It has been shown that the classification performance of DCCA is superior to that of CCA due to the discriminative power using class label information. On the other hand, Linear Discriminant Analysis (LDA) is a supervised dimension reduction method and it is known as a special case of CCA. In this paper, we analyze the relationship between DCCA and LDA, showing that the projective directions by DCCA are equal to the ones obtained from LDA with respect to an orthogonal transformation. Using the relation with LDA, we propose a new method that can enhance the performance of DCCA. The experimental results show that the proposed method exhibits better classification performance than the original DCCA.},
author = {Yong Joon Shin, Cheong Hee Park},
journal = {International Journal of Applied Mathematics and Computer Science},
keywords = {canonical correlation analysis; dimension reduction; discriminative canonical correlation analysis; feature fusion; linear discriminant analysis},
language = {eng},
number = {3},
pages = {549-558},
title = {Analysis of correlation based dimension reduction methods},
url = {http://eudml.org/doc/208069},
volume = {21},
year = {2011},
}

TY - JOUR
AU - Yong Joon Shin
AU - Cheong Hee Park
TI - Analysis of correlation based dimension reduction methods
JO - International Journal of Applied Mathematics and Computer Science
PY - 2011
VL - 21
IS - 3
SP - 549
EP - 558
AB - Dimension reduction is an important topic in data mining and machine learning. Especially dimension reduction combined with feature fusion is an effective preprocessing step when the data are described by multiple feature sets. Canonical Correlation Analysis (CCA) and Discriminative Canonical Correlation Analysis (DCCA) are feature fusion methods based on correlation. However, they are different in that DCCA is a supervised method utilizing class label information, while CCA is an unsupervised method. It has been shown that the classification performance of DCCA is superior to that of CCA due to the discriminative power using class label information. On the other hand, Linear Discriminant Analysis (LDA) is a supervised dimension reduction method and it is known as a special case of CCA. In this paper, we analyze the relationship between DCCA and LDA, showing that the projective directions by DCCA are equal to the ones obtained from LDA with respect to an orthogonal transformation. Using the relation with LDA, we propose a new method that can enhance the performance of DCCA. The experimental results show that the proposed method exhibits better classification performance than the original DCCA.
LA - eng
KW - canonical correlation analysis; dimension reduction; discriminative canonical correlation analysis; feature fusion; linear discriminant analysis
UR - http://eudml.org/doc/208069
ER -

References

top
  1. Anton, H. and Busby, R. (2003). Contemporary Linear Algebra, John Wiley and Sons, Denver, CO. Zbl1008.15001
  2. Baudat, G. and Anouar, F. (2000). Generalized discriminant analysis using a kernel approach, Neural Computation 12(10): 2385-2404. 
  3. Billings, S. and Lee, K. (2002). Nonlinear fisher discriminant analysis using a minimum squared error cost function and the orthogonal least squares algorithm, Neural Networks 15(2): 263-270. 
  4. Chen, L., Liao, H., M.Ko, Lin, J. and Yu, G. (2000). A new LDA-based face recognition system which can solve the small sample size problem, Pattern Recognition 33(10): 1713-1726. 
  5. Duda, R., Hart, P. and Stork, D. (2001). Pattern Classification, Wiley Interscience, New York, NY. Zbl0968.68140
  6. Fukunaga, K. (1990). Introduction to Statistical Pattern Recognition, 2nd Edn., Academic Press, San Diego, CA. Zbl0711.62052
  7. Fukunaga, K. and Mantock, J. (1983). Nonparametric discriminant analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence 5(6): 671-678. Zbl0542.62047
  8. Garthwaite, P. (1994). An interpretation of partial least squares, Journal of the American Statistical Society 89(425): 122-127. Zbl0793.62034
  9. He, X. and Niyogi, P. (2003). Locality preserving projections, Proceedings of the Advances in Neural Information Processing Systems Conference, Vancouver, Canada, pp. 153-160. 
  10. Hotelling, H. (1936). Relations between two sets of variates, Biometrika 28(3): 321-377. Zbl0015.40705
  11. Hou, C., Nie, F., Zhang, C. and Wu, Y. (2009). Learning an orthogonal and smooth subspace for image classification, IEEE Signal Processing Letters 16(4): 303-306. 
  12. Howland, P. and Park, H. (2004). Generalizing discriminant analysis using the generalized singular value decomposition, IEEE Transactions on Pattern Analysis and Machine Intelligence 26(8): 995-1006. 
  13. Jolliffe, I. (1986). Principal Component Analysis, Springer, New York, NY. Zbl1011.62064
  14. Nie, F., Xiang, S. and Zhang, C. (2007). Neighborhood minmax projections, Proceedings of the International Joint Conference on Artificial Intelligence, Hyderabad, India, pp. 993-998. 
  15. Pardalos, P. and Hansen, P. (2008). Data Mining and Mathematical Programming, CRM Proceedings & Lecture Notes, Vol. 45, American Mathematical Society, Montreal. 
  16. Park, C. and Park, H. (2008). A comparison of generalized linear discriminant analysis algorithms, Pattern Recognition 41(3): 1083-1097. Zbl1132.68656
  17. Roweis, S.T. and Saul, L.K. (2000). Nonlinear dimensionality reduction by locally linear embedding, Science 290(5500): 2323-2326. 
  18. Sugiyama, M. (2006). Local fisher discriminant analysis for supervised dimensionality reduction, Proceedings of the IEEE International Conference on Machine Learning, Pittsburgh, PA, USA, pp. 905-912. 
  19. Sun, Q., Zeng, S., Liu, Y., Heng, P. and Xia, D. (2005). A new method of feature fusion and its application in image recognition, Pattern Recognition 38(12): 2437-2448. 
  20. Sun, T. and Chen, S. (2007). Class label versus sample label-based CCA, Applied Mathematics and Computation 185(1): 272-283. Zbl1109.62053
  21. Sun, T., Chen, S., Yang, J. and Shi, P. (2008). A supervised combined feature extraction method for recognition, Procedings of the IEEE International Conference on Data Mining, Pisa, Italy, pp. 1043-1048. 
  22. Tenenbaum, J.B., de Silva, V. and Langford, J.C. (2000). A global geometric framework for nonlinear dimensionality reduction, Science 290(5500): 2319-2323. 
  23. Wegelin, J. (2000). A survey of partial least squares (PLS) methods, with emphasis on the two block case, Technical report, Department of Statistics, University of Washington, Seattle, WA. 
  24. Yan, S., Xu, D., Zhang, B., Zhang, H.-J., Yang, Q. and Lin, S. (2007). Graph embedding and extensions: A general framework for dimensionlity reduction, IEEE Transactions on Pattern Analysis and Machine Intelligence 29(1): 40-51. 
  25. Yang, J. and Yang, J.-Y. (2003). Why can LDA be performed in PCA transformed space?, Pattern Recognition 36(2): 563-566. 
  26. Yang, J., Yang, J., Zhang, D. and Lu, J. (2003). Feature fusion: Parallel strategy vs. serial strategy, Pattern Recognition 36(6): 1369-1381. Zbl1028.68152
  27. Ye, J. (2005). Characterization of a family of algorithms for generalized discriminant analysis on undersampled problems, Journal of Machine Learning Research 6(4): 483-502. Zbl1222.62081
  28. Yu, H. and Yang, J. (2001). A direct LDA algorithm for highdimensional data-with application to face recognition, Pattern Recognition 34(10): 2067-2070. Zbl0993.68091

Citations in EuDML Documents

top
  1. Rasa Karbauskaitė, Gintautas Dzemyda, Optimization of the maximum likelihood estimator for determining the intrinsic dimensionality of high-dimensional data
  2. Navdeep Goel, Kulbir Singh, A modified convolution and product theorem for the linear canonical transform derived by representation transformation in quantum mechanics
  3. Tomasz Górecki, Maciej Łuczak, Linear discriminant analysis with a generalization of the Moore-Penrose pseudoinverse
  4. Francisco A. Pujol, Higinio Mora, José A. Girona-Selva, A connectionist computational method for face recognition

NotesEmbed ?

top

You must be logged in to post comments.

To embed these notes on your page include the following JavaScript code on your page where you want the notes to appear.

Only the controls for the widget will be shown in your chosen language. Notes will be shown in their authored language.

Tells the widget how many notes to show per page. You can cycle through additional notes using the next and previous controls.

    
                

                
Note: Best practice suggests putting the JavaScript code just before the closing </body> tag.