Publications

Fast Solvers and Efficient Implementations for Distance Metric Learning

Sat, 3 May 2008 23:02:40 +0300

In this paper we study how to improve nearest neighbor classification by learning a Mahalanobis distance metric. We build on a recently proposed framework for distance metric learning known as large margin nearest neighbor (LMNN) classification.
Our paper makes three contributions. First, we describe a highly efficient solver for the particular instance of semidefinite programming that arises in LMNN classification; our solver can handle problems with billions of large margin constraints in a few hours. Second, we show how to reduce both training and testing times using metric ball trees; the speedups from ball trees are further magnified by learning low dimensional representations of the input space. Third, we show how to learn different Mahalanobis distance metrics in different parts of the input space. For large data sets, the use of locally adaptive distance metrics leads to even lower error rates.

Metric Learning with Convex Optimization (a PhD thesis)

Tue, 4 Sep 2007 23:58:09 +0300

Many machine learning algorithms rely heavily on the existence of a good measure of (dis-)similarity between input vectors. One of the most commonly used measures of dissimilarity is the Euclidean distance in input space. This is often suboptimal in many ways. The Euclidean distance metric does not incorporate any side-information that might be
available and it does not take advantage of the data structure or specifics of the machine learning goals. Ideally a metric should be learned for each specific task. Recent advances in numerical optimization provide us with a powerful tool for metric learning (and machine learning in general): Convex optimization. I will investigate two approaches to metric learning based on convex optimization for two different data scenarios:
The first algorithm, Large Margin Nearest Neighbor (LMNN), operates in a supervised scenario. LMNN learns a metric specifically to improve k-nearest neighbors classification.
This is achieved through a linear transformation of the input data that moves similarly labeled inputs close together and separates differently labeled inputs by a large margin. LMNN can be written as a semidefinite program that could be applied to large data sets with up to 60000 training samples. The second algorithm, Maximum Variance Unfolding (MVU), is designed for an unsupervised scenario. The algorithm finds a low dimensional Euclidean embedding of the data that preserves local distances while globally maximizing the variance. Similar to LMNN, MVU can also be phrased as a semidefinite program. This formulation gives
local guarantees and distinguishes the algorithm from prior work.

Metric Learning for Kernel Regression

Mon, 8 Jan 2007 03:47:39 +0200

K. Q. Weinberger, G. Tesauro (2007).
In Proceedings of the Eleventh International Workshop on Artificial Intelligence and Statistics (AISTATS-07), Puerto Rico.
[pdf] [bib]

Graph Laplacian Regularization for Large-Scale Semidefinite Programming

Wed, 6 Sep 2006 23:45:27 +0300

K. Q. Weinberger, F. Sha, Q. Zhu and L. K. Saul (2007).
In B. Schoelkopf, J. Platt, and T. Hofmann (eds.), Advances in Neural Information Processing Systems 19. MIT Press: Cambridge, MA.
[pdf] [bib]

An Introduction to Nonlinear Dimensionality Reduction by Maximum Variance Unfolding

Wed, 2 Aug 2006 06:51:47 +0300

K. Q. Weinberger and L. K. Saul (2006)
To appear in Proceedings of the National Conference on Artificial Intelligence (AAAI), Nectar paper, Boston MA
[pdf] [bib]

Unsupervised Learning of Image Manifolds by Semidefinite Programming

Mon, 3 Jul 2006 04:53:34 +0300

K. Q. Weinberger and L. K. Saul (2006)

International Journal of Computer Vision. Please download from www.springerlink.com. In Special Issue: Computer Vision and Pattern Recognition-CVPR 2004 Guest Editor(s): Aaron Bobick, Rama Chellappa, Larry Davis, Pages 77-90, Volume 70, Number 1, Springer Netherlands
[pdf][bib]

Spectral methods for dimensionality reduction

Tue, 16 May 2006 00:41:50 +0300

L. K. Saul, K. Q. Weinberger, J. H. Ham, F. Sha, and D.D. Lee (2006)
Spectral methods for dimensionality reduction
To appear in B. Schoelkopf, O. Chapelle, and A. Zien (eds.), Semisupervised Learning. MIT Press: Cambridge, MA.
[pdf][bib]

Distance Metric Learning for Large Margin Nearest Neighbor Classification

Tue, 10 Jan 2006 17:15:11 +0200

K. Q. Weinberger, J. Blitzer, and L. K. Saul (2006).
In Y. Weiss, B. Schoelkopf, and J. Platt (eds.), Advances in Neural Information Processing Systems 18. MIT Press: Cambridge, MA.
[pdf] [bib] [view talk]

Nonlinear Dimensionality Reduction by Semidefinite Programming and Kernel Matrix Factorization

Tue, 11 Jan 2005 04:48:27 +0200

K. Q. Weinberger, B. D. Packer and L. K. Saul (2005). (Outstanding student paper award)

In Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics (AISTATS-05), Barbados.
[pdf][bib] [view the talk]

Hierarchical Distributed Representations for Statistical Language Modeling

Thu, 6 Jan 2005 01:06:17 +0200

J. C. Blitzer, K. Q. Weinberger, L. K. Saul, F. C. N. Pereira (2005).
In Proceedings of the 18th annual conference on Neural Information Processing Systems (NIPS-04), Vancouver CA.
[pdf][bib]

Learning a kernel matrix for nonlinear dimensionality reduction

Sun, 4 Jul 2004 07:05:12 +0300

K. Q. Weinberger, F. Sha, and L. K. Saul (2004). (Outstanding student paper award)

In Proceedings of the Twenty First International Confernence on Machine Learning (ICML-04), Banff, Canada.
[pdf][bib] [view the talk]

Unsupervised learning of image manifolds by semidefinite programming

Fri, 2 Jul 2004 21:41:55 +0300

K. Q. Weinberger and L. K. Saul (2004). (Outstanding student paper award)
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR-04), Washington D.C.
[pdf] [abstract] [bib] [View the talk]