LEANN vector index: Set up low-storage RAG search

The LEANN repository benchmark, as reported by Marktechpost, shows a 60-million-chunk Wikipedia index shrinking from 201 GB to 6 GB. The LEANN vector index does this by discarding full embeddings after index construction and recomputing them at query time. Below is an installation and evaluation workflow for deciding whether that storage saving justifies slower search on your hardware.

The target is local document search and personal assistants, where an index can otherwise outweigh the documents themselves. The paper LEANN: A Low-Storage Vector Index puts conventional index overhead at 1.5 to 7 times raw data size. I'd shortlist LEANN when disk capacity blocks local retrieval, not as a default for every search service.

How the LEANN vector index avoids storing full vectors

The LEANN vector index still computes embeddings during index construction. The difference comes afterward: it discards the full vectors, retains a compact search structure, and regenerates selected embeddings when queries arrive.

The implementation builds on FAISS and Hierarchical Navigable Small World (HNSW) graphs. A graph search visits only part of the collection, so recomputation need not cover every document. Replacing every vector lookup with embedding inference would still be expensive.

LEANN combines three mechanisms to make the exchange practical:

• Two-level search limits exact recomputation. Product quantization (PQ) supplies inexpensive approximate distances. An approximate queue ranks candidates, while an exact queue receives promising candidates after their full embeddings are recomputed.
• Dynamic batching groups embedding work across search expansions instead of sending small batches to the GPU after each hop. The paper gives 64 as an example batch size for its A10 GPU, not a universal setting.
• High-degree-preserving pruning removes less useful graph connections while retaining important hubs. The researchers found that preserving the top 2% of high-degree nodes maintained high retrieval accuracy while reducing edges.

The distinction from pure compression matters. PQ guides exploration, but LEANN does not rely solely on compressed-vector distances for its final decisions. Exact recomputation restores information for selected candidates.

The search queue length, ef, remains a quality-versus-latency control: exploring more candidates can improve recall but takes longer. Graph pruning separately works within a storage budget, using offline profiling to select connection thresholds.

Read the benchmarks as separate experiments

The repository's 201 GB-to-6 GB comparison and the paper's results are not one interchangeable benchmark. Marktechpost's LEANN report attributes the former to the repository and a separate 4 GB result to the paper's v2 evaluation.

The original paper's reference datastore has the following configuration; these are experiment settings, not installation defaults.

The v1 evaluation uses an NVIDIA A10G server and an M1 Ultra Mac, with Llama-3.2-1B-Instruct for downstream generation. It reports index storage below 5% of raw text size and, for example, 90% recall on GPQA in under 2 seconds.

Marktechpost describes a later evaluation using an RTX 4090 workstation and Qwen3-4B. It reports 173 GB of embeddings plus a 15 GB HNSW graph for the 76 GB corpus. Keep those hardware, model and revision differences attached to the results. The later 4 GB figure should not silently replace the original paper's under-5% claim.

LEANN versus HNSW: the storage and latency exchange

Marktechpost attributes the following comparison to the research team's v2 paper; the measurements are self-reported. Retrieval timings target 90% Recall@3 on the RTX 4090.

Matching downstream accuracy does not mean matching retrieval speed. On NQ, the reported full pipeline increases from 20.95 seconds with HNSW to 23.34 seconds with LEANN, because generation already takes about 20 seconds.

The team evaluates NQ, TriviaQA, GPQA and HotpotQA. Recall@3 measures overlap with exact-search neighbors, not whether the generated answer is correct. The original paper separately evaluates answers with exact match and F1. That separation is useful: an index can reproduce nearest neighbors without guaranteeing a correct response.

Install LEANN and test local retrieval step by step

The installation command is available, but it does not ingest documents or create a searchable index. Choose a supported application and follow its indexing workflow.

1. Install the package. In an environment where uv is already available, this command installs LEANN.

   uv pip install leann

   Marktechpost lists Ubuntu, Arch, WSL, macOS and Windows support. A CPU-only Linux installation also exists, although the report gives no published CPU latency.

2. Select a small collection and its matching application. Ready-made workflows cover documents and browser history, among other personal-data sources. Ecatech's LEANN write-up recommends beginning with browser history or local documents before attempting larger collections.

   I'd start with documents whose useful passages I can identify manually, so the first trial tests retrieval rather than just whether installation succeeds.

3. Build the collection through the selected application's indexing workflow. LEANN's construction process embeds the items, creates the graph and then removes full embeddings while pruning graph storage. Keep the original content and embedding model available: query-time recomputation depends on them.

   The paper's embedding server is an on-device component using the original model. It is not inherently a requirement to send documents to an external embedding service.

4. Run representative searches and compare storage, retrieval time and answer quality. For this trial, I recommend recording retrieval separately from generation and checking returned passages against expected evidence. Use the same collection and queries when comparing another index.

   Do not treat a successful package installation as evidence that your workload meets its latency budget.

Where LEANN fits beyond document search

The repository includes applications for Apple Mail, Chrome history, WeChat, iMessage, and ChatGPT or Claude exports. Marktechpost also describes MCP readers for Slack and Twitter bookmarks, plus a semantic-search MCP server for Claude Code. Some applications, including Apple Mail search, are macOS-only.

The repository's personal-data comparisons report 400K chat messages going from 1.8 GB to 64 MB, and 38K browser entries from 130 MB to 6.4 MB. Repository examples illustrate storage savings on specific datasets; results vary with other data.

For coding retrieval, Marktechpost reports the team's evaluation on 30 SWE-Bench Pro tasks from ContextBench: initial relevant-code recall was 24.2% versus 11.4% for BM25, with 8.4% fewer agent tokens. That is a separate workload from Wikipedia question answering.

HNSW is the default backend, while DiskANN is offered for faster search. Generation can use Ollama, Hugging Face, Anthropic or an OpenAI-compatible API. Choosing a local index therefore remains separate from choosing where generation runs.

Recomputation still sets LEANN's latency floor

The LEANN vector index saves storage by doing work during retrieval; that work does not disappear. Marktechpost reports that embedding recomputation still accounts for about 76% of query latency, despite average speedups of 1.4 times from two-level search and 1.8 times with batching added.

The slow case deserves attention: its reported HotpotQA retrieval on the Mac takes 44.80 seconds, or 50.1% of total pipeline time. A fast GPQA result is not a general laptop latency guarantee.

Use LEANN for storage-constrained personal retrieval when measured delays are acceptable; keep a full-vector index that already fits and meets latency requirements, since the CPU-only installation does not establish CPU-only performance and a representative comparison should confirm the storage saving justifies the additional computation for latency-sensitive services.