Vector Indexes

You can create and manage multiple vector indexes on any Lance dataset. LanceDB offers two kinds of vector indexing algorithms: Inverted File (IVF) and Hierarchical Navigable Small World (HNSW).

​

Manual Indexing

If using LanceDB OSS, you will have to create the vector index manually, by calling table.create_index(), and updating the index as new data arrives and tuning its parameters is also a manual process.

​

Automatic Indexing

Enterprise-only Vector indexing is managed automatically in LanceDB Enterprise. As soon as data is updated, the system updates the index and optimizates it. This is done asynchronously as a background process. When you create a table in LanceDB Enterprise, LanceDB automatically:

• Infers the vector columns from the schema
• Create an optimized IVF_PQ index without manual configuration
• Automatically configure indexing parameters

The default distance is l2 (Euclidean). Use the same distance metric for index creation and search. Once a vector index exists, queries use the metric stored with that index. If you need to confirm an async build or refresh is finished, wait_for_index(...) waits for the named index to exist and for index_stats(...) to report num_unindexed_rows == 0; it can time out if new writes keep arriving.

​

Choose the Right Index

Use this table as a quick starting point for choosing the right index type and quantization method for your use case: Compression ratios are practical rules of thumb and can vary with vector distribution, metric, and configuration. For small dimensions, choose IVF_PQ for accuracy, not for guaranteed higher compression than IVF_RQ.

​

Index Tuning

Start with these values, then tune for your workload:

• HNSW-backed IVF indexes (IVF_HNSW_FLAT, IVF_HNSW_SQ, IVF_HNSW_PQ)
  • num_partitions: start at num_rows // 1,048,576 (rounded to an integer)
  • Lower num_partitions can reduce search latency, but index build may become slower because partitions are larger.
  • ef_construction: start at 150; increase for better recall, decrease for faster indexing.
• IVF_RQ
  • num_partitions: start at num_rows // 4096 (rounded to an integer). This is a strong default for most datasets.
• IVF_PQ
  • num_partitions: start at num_rows // 4096 (rounded to an integer).
  • num_sub_vectors: start at dimension // 8. Increase for better recall, decrease for faster search and smaller indexes.
  • For small dimensions (dimension <= 256), IVF_PQ is often preferred over IVF_RQ for better accuracy at similar query performance.

​

Example: Construct an IVF Index

In this example, we will create an index for a table containing 1536-dimensional vectors. The index will use IVF_PQ with L2 distance, which is well-suited for high-dimensional vector search. Make sure you have enough data in your table (at least a few thousand rows) for effective index training.

​

Index Configuration

Sometimes you need to configure the index beyond default parameters:

• Index Types:
  • IVF_HNSW_FLAT: highest recall, with no vector quantization
  • IVF_HNSW_SQ: best recall/latency trade-off
  • IVF_RQ: best compression for large, high-dimensional datasets
  • IVF_PQ: often higher accuracy than IVF_RQ for small dimensions (<= 256) at similar query performance
• metrics: default is l2, other available are cosine or dot
  • When using cosine similarity, distances range from 0 (identical vectors) to 2 (maximally dissimilar)
• num_partitions: use index-specific starting points from the section above:
  • HNSW-backed IVF indexes (IVF_HNSW_FLAT, IVF_HNSW_SQ, IVF_HNSW_PQ): num_rows // 1,048,576
  • IVF_RQ and IVF_PQ: num_rows // 4096
• num_sub_vectors: applies to IVF_PQ; start with dimension // 8. Larger values often improve recall but can slow search.

Let’s take a look at a sample request for an IVF index:

​

1. Setup

Connect to LanceDB and open the table you want to index.

​

2. Construct an IVF Index

Create an IVF_PQ index with cosine similarity. Specify vector_column_name if you use multiple vector columns or non-default names. For a vector field nested inside a struct, use dot notation (e.g. image.embedding); see Selecting the vector column for the full syntax. You can switch index_type to IVF_RQ, IVF_HNSW_SQ, or IVF_HNSW_FLAT depending on your recall/latency/compression target.

​

Indexing nested vector fields

If your vector column lives inside a struct, pass its full dotted path as vector_column_name. The same path is used at query time and is what list_indices() reports under columns:

​

Async API and Config Objects

With asynchronous Python connections, create vector indexes with await table.create_index("vector", config=...). The config object carries the same index choices you configure in the synchronous API, such as distance metric, partition count, and quantization settings: Use these Python config classes for the index types shown on this page:

​

3. Query the IVF Index

Search using a random 1,536-dimensional embedding.

​

Search Configuration

Core knobs available on a vector search call: Recommended nprobes behavior by index type:

​

Advanced Search Controls

These controls are useful for thresholded retrieval, recall measurement, and working around index-level metric constraints. Thresholding with distance_range: Measuring recall with bypass_vector_index: Compare ANN results against a flat-scan ground truth to compute recall@k. This is the standard way to pick nprobes for your workload.

​

Example: Construct an HNSW Index

​

Index Configuration

There are four key parameters to set when constructing an HNSW index:

• index_type: choose IVF_HNSW_SQ for a strong recall/latency/size trade-off, or IVF_HNSW_FLAT when you want the IVF+HNSW structure without vector quantization.
• metric: The default is l2 euclidean distance metric. Other available are dot and cosine.
• m: The number of neighbors to select for each vector in the HNSW graph.
• ef_construction: The number of candidates to evaluate during the construction of the HNSW graph.

​

1. Construct an HNSW Index

The snippet below uses IVF_HNSW_SQ. If you want the unquantized variant, change index_type to IVF_HNSW_FLAT.

​

2. Query the HNSW Index

​

Example: Construct a Binary Vector Index

Binary vectors are useful for hash-based retrieval, fingerprinting, or any scenario where data can be represented as bits.

​

Index Configuration

• Store binary vectors as fixed-size binary data (uint8 arrays, with 8 bits per byte). For storage, pack binary vectors into bytes to save space.
• Index Type: IVF_FLAT is used for indexing binary vectors
• metric: the hamming distance is used for similarity search
• The dimension of binary vectors must be a multiple of 8. For example, a 128-dimensional vector is stored as a uint8 array of size 16.

​

1. Create Table and Schema

​

2. Generate and Add Data

​

3. Construct the Binary Index

​

4. Vector Search

​

Check Index Status

Vector index creation runs in the background and may take some time to complete. While it is ongoing, you can check its status either programmatically through the API or from the LanceDB Enterprise UI. In the LanceDB Enterprise UI, navigate to your table page - the “Index” column reflects each column’s index status: it is blank when no index exists, shows an “in progress” label while the index is being built, and shows the index type once the build completes. Programmatically, use list_indices() and index_stats(). By default, the index name is formed by appending _idx to the column name (e.g., a keywords_embeddings column produces keywords_embeddings_idx). Note that list_indices() only returns information after the index is fully built. To wait until all data is fully indexed, you can specify the wait_timeout parameter on create_index() or call wait_for_index() on the table. Each entry returned by list_indices() also carries detailed per-index metadata, so you can inspect an index without a follow-up index_stats() call. Node.js exposes the same fields in camelCase (num_indexed_rows → numIndexedRows):

​

Custom Index Names

The {column}_idx suffix is a default convention, not the only supported naming path. Pass name=... to create_index() to override it — useful when you want to manage multiple indexes on the same column (for example, side-by-side IVF_PQ and IVF_HNSW_SQ builds) or when you script index replacement by name. Once set, list_indices(), index_stats(name), and wait_for_index([name]) all reference the custom name.