Quality and efficiency in high dimensional Nearest neighbor search

Yufei Tao, Ke Yi, Cheng Sheng, Panos Kalnis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

161 Scopus citations

Abstract

Nearest neighbor (NN) search in high dimensional space is an important problem in many applications. Ideally, a practical solution (i) should be implementable in a relational database, and (ii) its query cost should grow sub-linearly with the dataset size, regardless of the data and query distributions. Despite the bulk of NN literature, no solution fulfills both requirements, except locality sensitive hashing (LSH). The existing LSH implementations are either rigorous or adhoc. Rigorous-LSH ensures good quality of query results, but requires expensive space and query cost. Although adhoc-LSH is more efficient, it abandons quality control, i.e., the neighbor it outputs can be arbitrarily bad. As a result, currently no method is able to ensure both quality and efficiency simultaneously in practice. Motivated by this, we propose a new access method called the locality sensitive B-tree (LSB-tree) that enables fast highdimensional NN search with excellent quality. The combination of several LSB-trees leads to a structure called the LSB-forest that ensures the same result quality as rigorous-LSH, but reduces its space and query cost dramatically. The LSB-forest also outperforms adhoc-LSH, even though the latter has no quality guarantee. Besides its appealing theoretical properties, the LSB-tree itself also serves as an effective index that consumes linear space, and supports efficient updates. Our extensive experiments confirm that the LSB-tree is faster than (i) the state of the art of exact NN search by two orders of magnitude, and (ii) the best (linear-space) method of approximate retrieval by an order of magnitude, and at the same time, returns neighbors with much better quality. © 2009 ACM.
Original languageEnglish (US)
Title of host publicationProceedings of the 35th SIGMOD international conference on Management of data - SIGMOD '09
PublisherAssociation for Computing Machinery (ACM)
Pages563-575
Number of pages13
ISBN (Print)9781605585543
DOIs
StatePublished - 2009

Fingerprint

Dive into the research topics of 'Quality and efficiency in high dimensional Nearest neighbor search'. Together they form a unique fingerprint.

Cite this