Many machine learning algorithms rely heavily on the existence of a good measure of (dis-)similarity between input vectors. One of the most commonly used measures of dissimilarity is the Euclidean distance in input space. This is often suboptimal in many ways. The Euclidean distance metric does not incorporate any side-information that might be

available and it does not take advantage of the data structure or specifics of the machine learning goals. Ideally a metric should be learned for each specific task. Recent advances in numerical optimization provide us with a powerful tool for metric learning (and machine learning in general): Convex optimization. I will investigate two approaches to metric learning based on convex optimization for two different data scenarios:

The first algorithm, Large Margin Nearest Neighbor (LMNN), operates in a supervised scenario. LMNN learns a metric specifically to improve k-nearest neighbors classification.

This is achieved through a linear transformation of the input data that moves similarly labeled inputs close together and separates differently labeled inputs by a large margin. LMNN can be written as a semidefinite program that could be applied to large data sets with up to 60000 training samples. The second algorithm, Maximum Variance Unfolding (MVU), is designed for an unsupervised scenario. The algorithm finds a low dimensional Euclidean embedding of the data that preserves local distances while globally maximizing the variance. Similar to LMNN, MVU can also be phrased as a semidefinite program. This formulation gives

local guarantees and distinguishes the algorithm from prior work.

 

K. Q. Weinberger, G. Tesauro (2007).

In Proceedings of the Eleventh International Workshop on Artificial Intelligence and Statistics (AISTATS-07), Puerto Rico.

K. Q. Weinberger, F. Sha, Q. Zhu and L. K. Saul (2007).

In B. Schoelkopf, J. Platt, and T. Hofmann (eds.), Advances in Neural Information Processing Systems 19. MIT Press: Cambridge, MA.

K. Q. Weinberger and L. K. Saul (2006)

To appear in Proceedings of the National Conference on Artificial Intelligence (AAAI), Nectar paper, Boston MA

K. Q. Weinberger and L. K. Saul (2006)

 

 

L. K. Saul, K. Q. Weinberger, J. H. Ham, F. Sha, and D.D. Lee (2006)

Spectral methods for dimensionality reduction

To appear in B. Schoelkopf, O. Chapelle, and A. Zien (eds.), Semisupervised Learning. MIT Press: Cambridge, MA.

K. Q. Weinberger, J. Blitzer, and L. K. Saul (2006).

In Y. Weiss, B. Schoelkopf, and J. Platt (eds.), Advances in Neural Information Processing Systems 18. MIT Press: Cambridge, MA.

K. Q. Weinberger, B. D. Packer and L. K. Saul (2005).  (Outstanding student paper award)

 

In Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics (AISTATS-05), Barbados.

J. C. Blitzer, K. Q. Weinberger, L. K. Saul, F. C. N. Pereira (2005).

In Proceedings of the 18th annual conference on Neural Information Processing Systems (NIPS-04), Vancouver CA.

K. Q. Weinberger, F. Sha, and L. K. Saul (2004). (Outstanding student paper award)

 

In Proceedings of the Twenty First International Confernence on Machine Learning (ICML-04), Banff, Canada.

K. Q. Weinberger and L. K. Saul (2004). (Outstanding student paper award)

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR-04), Washington D.C.