barrel_vectordb

v2.1.1

README

# barrel_vectordb

**High-performance vector database for Erlang with HNSW, FAISS, DiskANN, and BM25 backends**

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)

[Documentation](https://barrel-db.eu/docs/lib/vectordb/) | [Examples](./examples) | [barrel-db.eu](https://barrel-db.eu)

Quick Start

With Embeddings

%% Start a store with local Python embeddings
{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    embedder => {local, #{}}  %% requires Python + sentence-transformers
}).

%% Add documents (text is embedded automatically)
ok = barrel_vectordb:add(my_store, <<"doc1">>, <<"Hello world">>, #{}).
ok = barrel_vectordb:add(my_store, <<"doc2">>, <<"Goodbye world">>, #{}).

%% Search with text query
{ok, Results} = barrel_vectordb:search(my_store, <<"hi there">>, #{k => 5}).
%% => [#{key => <<"doc1">>, text => <<"Hello world">>, score => 0.89, ...}, ...]

Vector-Only (no embedder)

%% Start a store without embedder
{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    dimensions => 768
}).

%% Add with pre-computed vectors
ok = barrel_vectordb:add_vector(my_store, <<"doc1">>, <<"Hello">>, #{}, Vector).

%% Search with vector query
{ok, Results} = barrel_vectordb:search_vector(my_store, QueryVector, #{k => 5}).

Installation

Add to your rebar.config:

{deps, [
    {barrel_vectordb, "2.1.1"}
]}.

This includes barrel_embed for embedding support. To use text-based operations (add/4, search/3), configure an embedder provider. Without an embedder configured, use add_vector/5 and search_vector/3 with pre-computed vectors.

Optional: Reranking

For cross-encoder reranking, add barrel_rerank:

{deps, [
    {barrel_vectordb, "2.1.1"},
    {barrel_rerank, "1.0.0"}
]}.

Core API

Add Documents

%% Add with text (requires embedder)
ok = barrel_vectordb:add(Store, Id, Text, Metadata).

%% Add with explicit vector (no embedder required)
ok = barrel_vectordb:add_vector(Store, Id, Text, Metadata, Vector).

%% Add batch (requires embedder)
{ok, #{inserted := N}} = barrel_vectordb:add_batch(Store, [
    {<<"id1">>, <<"text 1">>, #{type => a}},
    {<<"id2">>, <<"text 2">>, #{type => b}}
]).

Search

%% Search with text query (requires embedder)
{ok, Results} = barrel_vectordb:search(Store, <<"query text">>, #{k => 10}).

%% Search with vector (no embedder required)
{ok, Results} = barrel_vectordb:search_vector(Store, Vector, #{k => 10}).

%% Search with metadata filter
{ok, Results} = barrel_vectordb:search(Store, <<"query">>, #{
    k => 10,
    filter => fun(Meta) -> maps:get(type, Meta) =:= important end
}).

%% Search with optimized options (skip text/metadata for faster results)
{ok, Results} = barrel_vectordb:search_vector(Store, Vector, #{
    k => 50,
    include_text => false,      %% Skip text lookup
    include_metadata => false   %% Skip metadata lookup
}).

%% Search with custom ef_search (higher = better recall, slower)
{ok, Results} = barrel_vectordb:search_vector(Store, Vector, #{
    k => 10,
    ef_search => 200   %% Default is max(k, 50)
}).

Document Operations

%% Get document by ID
{ok, Doc} = barrel_vectordb:get(Store, <<"doc1">>).

%% Update document (requires embedder)
ok = barrel_vectordb:update(Store, <<"doc1">>, <<"New text">>, #{}).

%% Upsert (requires embedder)
ok = barrel_vectordb:upsert(Store, <<"doc1">>, <<"Text">>, #{}).

%% Delete
ok = barrel_vectordb:delete(Store, <<"doc1">>).

%% Peek (sample documents)
{ok, Docs} = barrel_vectordb:peek(Store, 10).

%% Count
N = barrel_vectordb:count(Store).

%% Checkpoint HNSW index (speeds up restart)
ok = barrel_vectordb:checkpoint(Store).

Configuration

barrel_vectordb:start_link(#{
    name => my_store,              %% Store name (required)
    path => "/var/data/vectors",   %% RocksDB path
    dimensions => 768,             %% Vector dimensions (default: 768)
    backend => hnsw,               %% Index backend: hnsw (default) or faiss
    embedder => EmbedderConfig,    %% Embedding provider (optional)
    hnsw => #{                     %% HNSW index parameters
        m => 16,
        ef_construction => 200
    },
    batch => #{                    %% Write batching options
        min_batch_size => 4,       %% Min requests before batching
        max_batch_size => 256      %% Max batch size
    }
}).

Index Backends

barrel_vectordb supports two vector index backends:

HNSW (Default)

Pure Erlang HNSW implementation. No external dependencies.

{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    backend => hnsw  %% default, can be omitted
}).

FAISS

High-performance FAISS backend via barrel_faiss NIF. Typically 2-6x faster than pure Erlang HNSW for insert and search operations.

Installation:

Add to your rebar.config:

{profiles, [
    {faiss, [
        {deps, [
            {barrel_faiss, {git, "https://github.com/barrel-db/barrel.git", {branch, "main"}}}
        ]}
    ]}
]}.

Requires FAISS library installed on your system. See barrel_faiss README for installation instructions.

Usage:

{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    backend => faiss,
    faiss => #{
        index_type => <<"HNSW32">>,  %% default
        distance_fn => cosine        %% cosine (default) or euclidean
    }
}).

Backend Comparison:

Feature | HNSW | FAISS
Dependencies | None | barrel_faiss NIF
Insert speed | Baseline | 1.6-3x faster
Search speed | Baseline | 2x faster
Index build | Baseline | 6x faster
Delete speed | Fast (native) | Slower (soft delete)
Memory | Higher | Lower

When to use FAISS:

Large indexes (>100K vectors)

High insert throughput requirements

Search latency is critical

When to use HNSW:

Simpler deployment (no NIF)

Frequent deletions

Smaller indexes

Vector Quantization

Reduce memory usage with TurboQuant compression:

%% Create quantizer (no training needed)
{ok, TQ} = barrel_vectordb_turboquant:new(#{
    dimension => 768,
    bits => 3
}).

%% Encode vector (768 floats -> ~388 bytes)
Code = barrel_vectordb_turboquant:encode(TQ, Vector).

%% Fast distance computation
Tables = barrel_vectordb_turboquant:precompute_tables(TQ, Query),
Distance = barrel_vectordb_turboquant:distance_nif(Tables, Code).

For large dimensions, use Subspace-TurboQuant:

{ok, TQS} = barrel_vectordb_turboquant_subspace:new(#{
    dimension => 1536,
    m => 16  %% 16 subspaces
}).

See TurboQuant Documentation for details.

Embedding Providers

Embedder is explicit - if not configured, only add_vector/5 and search_vector/3 work. Text-based operations return {error, embedder_not_configured}.

Local

Local Python with sentence-transformers. CPU-based, no external API calls.

embedder => {local, #{
    python => "python3",                %% Python executable (default)
    model => "BAAI/bge-base-en-v1.5",   %% Model name (default, 768 dims)
    timeout => 120000                   %% Timeout in ms (default)
}}

Setup with virtual environment (recommended):

# Create virtual environment
python3 -m venv ~/.venv/barrel_embed
source ~/.venv/barrel_embed/bin/activate

# Install dependencies
pip install sentence-transformers

# Verify installation
python -c "from sentence_transformers import SentenceTransformer; print('OK')"

Then start the store with the virtual environment's Python path:

{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    embedder => {local, #{
        python => "/home/user/.venv/barrel_embed/bin/python"
    }}
}).

Or activate the venv before starting your Erlang application:

source ~/.venv/barrel_embed/bin/activate
rebar3 shell

%% Now python3 will use the venv automatically
{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    embedder => {local, #{}}  %% uses default python3
}).

Supported Models:

Any sentence-transformers or HuggingFace model works. Popular choices:

Model | Dimensions | Notes
BAAI/bge-base-en-v1.5 | 768 | Default, good quality/speed
BAAI/bge-small-en-v1.5 | 384 | Faster, smaller
BAAI/bge-large-en-v1.5 | 1024 | Best quality, slower
sentence-transformers/all-MiniLM-L6-v2 | 384 | Fast, general purpose
sentence-transformers/all-mpnet-base-v2 | 768 | High quality
nomic-ai/nomic-embed-text-v1.5 | 768 | Long context (8192 tokens)

The dimension is auto-detected from the model.

Ollama

Local Ollama server. Requires Ollama to be running.

embedder => {ollama, #{
    url => <<"http://localhost:11434">>,   %% Ollama API URL (default)
    model => <<"nomic-embed-text">>,       %% Model name (default, 768 dims)
    timeout => 30000                       %% Timeout in ms (default)
}}

# Pull embedding models:
ollama pull nomic-embed-text

Supported Models:

Model | Dimensions | Notes
nomic-embed-text | 768 | Default, general purpose
mxbai-embed-large | 1024 | High quality
all-minilm | 384 | Fast
snowflake-arctic-embed | 1024 | Multilingual

OpenAI

OpenAI Embeddings API. Requires an API key.

embedder => {openai, #{
    api_key => <<"sk-...">>,               %% API key (or set OPENAI_API_KEY env var)
    model => <<"text-embedding-3-small">>, %% Model name (default, 1536 dims)
    timeout => 30000                       %% Timeout in ms (default)
}}

# Set API key as environment variable (alternative to config)
export OPENAI_API_KEY=sk-...

Supported Models:

Model | Dimensions | Notes
text-embedding-3-small | 1536 | Default, fast and cheap
text-embedding-3-large | 3072 | Higher quality
text-embedding-ada-002 | 1536 | Legacy model

FastEmbed

Lightweight ONNX-based embeddings. Faster than sentence-transformers for many models.

embedder => {fastembed, #{
    python => "python3",                    %% Python executable (default)
    model => "BAAI/bge-small-en-v1.5",      %% Model name (default, 384 dims)
    timeout => 120000                       %% Timeout in ms (default)
}}

Setup:

pip install fastembed

Supported Models:

Model | Dimensions | Notes
BAAI/bge-small-en-v1.5 | 384 | Default, fast
BAAI/bge-base-en-v1.5 | 768 | Good balance
sentence-transformers/all-MiniLM-L6-v2 | 384 | General purpose

Provider Chain

Try providers in order until one succeeds.

embedder => [
    {openai, #{api_key => <<"sk-...">>}},  %% Try OpenAI first
    {ollama, #{url => <<"http://localhost:11434">>}},
    {local, #{}}  %% Fallback to CPU
]

Advanced Embedding Types

SPLADE (Sparse Embeddings)

Neural sparse embeddings with term expansion. Produces sparse vectors for hybrid search.

%% Initialize SPLADE provider
{ok, State} = barrel_embed:init(#{
    embedder => {splade, #{
        model => "prithivida/Splade_PP_en_v1"
    }}
}).

%% Get sparse vectors directly
{ok, SparseVec} = barrel_embed_splade:embed_sparse(<<"query text">>, Config).
%% => #{indices => [1, 42, 156], values => [0.5, 0.3, 0.8]}

Setup:

pip install transformers torch

ColBERT (Late Interaction)

Multi-vector embeddings for fine-grained token-level matching.

%% Initialize ColBERT provider
{ok, State} = barrel_embed:init(#{
    embedder => {colbert, #{
        model => "colbert-ir/colbertv2.0"
    }}
}).

%% Get multi-vector embeddings
{ok, MultiVec} = barrel_embed_colbert:embed_multi(<<"query text">>, Config).
%% => [[0.1, 0.2, ...], [0.3, 0.4, ...], ...]  %% One vector per token

%% MaxSim scoring between query and document
Score = barrel_embed_colbert:maxsim_score(QueryVecs, DocVecs).

Setup:

pip install transformers torch

CLIP (Image Embeddings)

Cross-modal embeddings for image-text search. Images and text share the same vector space.

%% Initialize CLIP provider
{ok, State} = barrel_embed:init(#{
    embedder => {clip, #{
        model => "openai/clip-vit-base-patch32"
    }}
}).

%% Embed text (for cross-modal search)
{ok, TextVec} = barrel_embed_clip:embed(<<"a photo of a cat">>, Config).

%% Embed image (base64 encoded)
{ok, ImageVec} = barrel_embed_clip:embed_image(Base64Image, Config).

%% TextVec and ImageVec are in the same space - compare with cosine similarity!

Setup:

pip install transformers torch pillow

Supported Models:

Model | Dimensions | Notes
openai/clip-vit-base-patch32 | 512 | Default, fast
openai/clip-vit-base-patch16 | 512 | Higher quality
openai/clip-vit-large-patch14 | 768 | Best quality

Reranking

Requires: barrel_rerank dependency

Cross-encoder reranking for improved search relevance. Use after initial vector search.

%% Add barrel_rerank to your deps
{deps, [
    {barrel_vectordb, "2.1.1"},
    {barrel_embed, "2.3.0"},
    {barrel_rerank, "1.0.0"}
]}.

%% Start the reranker
{ok, Reranker} = barrel_rerank:start_link(#{
    model => "cross-encoder/ms-marco-MiniLM-L-6-v2"
}).

%% Two-stage retrieval
%% Stage 1: Fast vector search (top 100)
{ok, Candidates} = barrel_vectordb:search(Store, Query, #{k => 100}).

%% Stage 2: Rerank candidates
Docs = [maps:get(text, C) || C <- Candidates],
{ok, Ranked} = barrel_rerank:rerank(Reranker, Query, Docs).
%% => [{0, 0.95}, {2, 0.82}, {1, 0.45}, ...]  %% {Index, Score}

%% Get top 10 after reranking
Top10 = [lists:nth(Idx + 1, Candidates) || {Idx, _} <- lists:sublist(Ranked, 10)].

%% Cleanup
ok = barrel_rerank:stop(Reranker).

Setup:

The venv with dependencies is auto-created on first use, or manually:

{ok, _} = barrel_rerank_venv:ensure_venv().

Supported Models:

Model | Notes
cross-encoder/ms-marco-MiniLM-L-6-v2 | Default, fast
cross-encoder/ms-marco-MiniLM-L-12-v2 | Better quality
BAAI/bge-reranker-base | Good quality
BAAI/bge-reranker-large | Best quality

BM25 Sparse Retrieval

Pure Erlang BM25 implementation for lexical search. In-memory index.

%% Create index
Index = barrel_vectordb_bm25:new().

%% Add documents
Index1 = barrel_vectordb_bm25:add(Index, <<"doc1">>, <<"The quick brown fox">>).
Index2 = barrel_vectordb_bm25:add(Index1, <<"doc2">>, <<"The lazy dog">>).

%% Search
Results = barrel_vectordb_bm25:search(Index2, <<"quick fox">>, 10).
%% => [{<<"doc1">>, 2.45}, ...]

%% Get sparse vector for a document
{ok, SparseVec} = barrel_vectordb_bm25:get_vector(Index2, <<"doc1">>).
%% => #{indices => [hash1, hash2, ...], values => [1.2, 0.8, ...]}

%% Index stats
Stats = barrel_vectordb_bm25:stats(Index2).
%% => #{doc_count => 2, avg_doc_len => 3.5, ...}

Note: BM25 index is in-memory and not persisted. Rebuild from documents on startup.

Search Options

Option | Default | Description
k | 5 | Number of results to return
filter | - | Function fun(Metadata) -> boolean() to filter results
include_text | true | Include text in results
include_metadata | true | Include metadata in results
ef_search | max(k, 50) | Search width (higher = better recall, slower)

HNSW Parameters

Parameter | Default | Description
m | 16 | Max connections per node
ef_construction | 200 | Build-time search width
ef_search | 50 | Default query-time search width
distance_fn | cosine | cosine or euclidean

Testing

Unit Tests

Unit tests use mocking and don't require external dependencies:

rebar3 eunit

With Optional Dependencies

To run tests that exercise barrel_embed:

rebar3 as test_embed eunit

To run tests with all optional dependencies:

rebar3 as test_full eunit

Performance

Search Latency

Metric | Typical Value
P50 | ~1ms
P99 | ~5ms

Optimizations

Batch writes: Concurrent writes are automatically batched via gen_batch_server

Batch lookups: Search uses rocksdb:multi_get for efficient result fetching

Skip options: Use include_text => false to skip unnecessary RocksDB reads

HNSW optimization: O(log N) candidate management with balanced trees

Benchmarking

Run the benchmark suite:

rebar3 as bench compile && rebar3 as bench eunit --module=barrel_vectordb_bench

Backend Comparison Benchmarks

Compare HNSW vs FAISS performance:

# Quick comparison
./scripts/run_backend_bench.sh --quick

# Default comparison
./scripts/run_backend_bench.sh

# Full benchmark suite
./scripts/run_backend_bench.sh --full

Or programmatically:

rebar3 as bench_faiss shell
barrel_vectordb_backend_bench:run_all().

Architecture

Storage: RocksDB with column families

Index: HNSW for approximate nearest neighbor search

Vectors: 8-bit quantization with norm caching

Embeddings: Pluggable providers with fallback

Batching: gen_batch_server for automatic write coalescing

See the API documentation for detailed architecture information.

Support

Channel | For
GitHub Issues | Bug reports, feature requests
Email | Commercial inquiries

License

Apache-2.0. See LICENSE for details.

Built by Enki Multimedia | barrel-db.eu

Quick Start

Get started with Barrel VectorDB in a few minutes.

Installation

Add to your rebar.config:

{deps, [
    {barrel_vectordb, {git, "https://github.com/barrel-db/barrel_vectordb.git", {branch, "main"}}}
]}.

Basic Usage

With Embeddings

If you want automatic text embedding, configure an embedder:

%% Start a store with local Python embeddings
{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    embedder => {local, #{}}  %% requires Python + sentence-transformers
}).

%% Add documents (text is embedded automatically)
ok = barrel_vectordb:add(my_store, <<"doc1">>, <<"Hello world">>, #{}).
ok = barrel_vectordb:add(my_store, <<"doc2">>, <<"Goodbye world">>, #{}).

%% Search with text query
{ok, Results} = barrel_vectordb:search(my_store, <<"hi there">>, #{k => 5}).
%% => [#{key => <<"doc1">>, text => <<"Hello world">>, score => 0.89, ...}, ...]

Vector-Only Mode

If you have pre-computed vectors, skip the embedder:

%% Start a store without embedder
{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    dimensions => 768
}).

%% Add with pre-computed vectors
ok = barrel_vectordb:add_vector(my_store, <<"doc1">>, <<"Hello">>, #{}, Vector).

%% Search with vector query
{ok, Results} = barrel_vectordb:search_vector(my_store, QueryVector, #{k => 5}).

Core API

Add Documents

%% Add with text (requires embedder)
ok = barrel_vectordb:add(Store, Id, Text, Metadata).

%% Add with explicit vector (no embedder required)
ok = barrel_vectordb:add_vector(Store, Id, Text, Metadata, Vector).

%% Add batch (requires embedder)
{ok, #{inserted := N}} = barrel_vectordb:add_batch(Store, [
    {<<"id1">>, <<"text 1">>, #{type => a}},
    {<<"id2">>, <<"text 2">>, #{type => b}}
]).

Search

%% Search with text query (requires embedder)
{ok, Results} = barrel_vectordb:search(Store, <<"query text">>, #{k => 10}).

%% Search with vector (no embedder required)
{ok, Results} = barrel_vectordb:search_vector(Store, Vector, #{k => 10}).

%% Search with metadata filter
{ok, Results} = barrel_vectordb:search(Store, <<"query">>, #{
    k => 10,
    filter => fun(Meta) -> maps:get(type, Meta) =:= important end
}).

%% Search with optimized options (skip text/metadata for faster results)
{ok, Results} = barrel_vectordb:search_vector(Store, Vector, #{
    k => 50,
    include_text => false,
    include_metadata => false
}).

Document Operations

%% Get document by ID
{ok, Doc} = barrel_vectordb:get(Store, <<"doc1">>).

%% Update document (requires embedder)
ok = barrel_vectordb:update(Store, <<"doc1">>, <<"New text">>, #{}).

%% Upsert (requires embedder)
ok = barrel_vectordb:upsert(Store, <<"doc1">>, <<"Text">>, #{}).

%% Delete
ok = barrel_vectordb:delete(Store, <<"doc1">>).

%% Count
N = barrel_vectordb:count(Store).

%% Checkpoint HNSW index (speeds up restart)
ok = barrel_vectordb:checkpoint(Store).

Configuration

barrel_vectordb:start_link(#{
    name => my_store,              %% Store name (required)
    path => "/var/data/vectors",   %% RocksDB path
    dimensions => 768,             %% Vector dimensions (default: 768)
    backend => hnsw,               %% Index backend: hnsw (default), faiss, or diskann
    embedder => EmbedderConfig,    %% Embedding provider (optional)
    hnsw => #{                     %% HNSW index parameters
        m => 16,
        ef_construction => 200
    },
    batch => #{                    %% Write batching options
        min_batch_size => 4,
        max_batch_size => 256
    }
}).

Search Options

Option | Default | Description
k | 5 | Number of results to return
filter | - | Function fun(Metadata) -> boolean() to filter results
include_text | true | Include text in results
include_metadata | true | Include metadata in results
ef_search | max(k, 50) | Search width (higher = better recall, slower)

HNSW Parameters

Parameter | Default | Description
m | 16 | Max connections per node
ef_construction | 200 | Build-time search width
ef_search | 50 | Default query-time search width
distance_fn | cosine | cosine or euclidean

Index Backends

HNSW (Default)

Pure Erlang HNSW implementation. No external dependencies.

{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    backend => hnsw
}).

FAISS

High-performance FAISS backend via barrel_faiss NIF.

{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    backend => faiss,
    faiss => #{
        index_type => <<"HNSW32">>,
        distance_fn => cosine
    }
}).

Feature | HNSW | FAISS
Dependencies | None | barrel_faiss NIF
Insert speed | Baseline | 1.6-3x faster
Search speed | Baseline | 2x faster
Delete speed | Fast (native) | Slower (soft delete)

DiskANN

SSD-optimized Vamana graph for billion-scale vector search with minimal memory.

{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    backend => diskann,
    diskann => #{
        r => 64,                    %% Max out-degree
        l_build => 100,             %% Build search width
        l_search => 100,            %% Query search width
        storage_mode => disk,       %% memory | disk
        hot_enabled => true         %% Enable hot layer for fast writes
    }
}).

Feature | HNSW | FAISS | DiskANN
Dependencies | None | barrel_faiss NIF | None
Memory usage | ~4KB/vector | Medium | ~530B/vector (disk mode)
Insert speed | Fast | 1.6-3x faster | Slower (batch optimized)
Search speed | ~0.25ms | 2x faster | ~1ms
Best for | Small-medium | GPU, batch queries | Large-scale, memory-constrained

See DiskANN documentation for detailed configuration.

Next Steps

Embedding Models - Configure embedding providers

BM25 Text Search - Lexical and hybrid search

API reference - Full API reference

Embedding Models

Barrel VectorDB supports multiple embedding providers through the barrel_embed library (included as a dependency).

The embedder is explicit - if not configured, only add_vector/5 and search_vector/3 work. Text-based operations return {error, embedder_not_configured}.

Providers

Local (sentence-transformers)

Local Python with sentence-transformers. CPU-based, no external API calls.

embedder => {local, #{
    python => "python3",                %% Python executable (default)
    model => "BAAI/bge-base-en-v1.5",   %% Model name (default, 768 dims)
    timeout => 120000                   %% Timeout in ms (default)
}}

Setup with virtual environment (recommended):

# Create virtual environment
python3 -m venv ~/.venv/barrel_embed
source ~/.venv/barrel_embed/bin/activate

# Install dependencies
pip install sentence-transformers

# Verify installation
python -c "from sentence_transformers import SentenceTransformer; print('OK')"

Then configure the store:

{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/tmp/vectors",
    embedder => {local, #{
        python => "/home/user/.venv/barrel_embed/bin/python"
    }}
}).

Supported Models:

Model | Dimensions | Notes
BAAI/bge-base-en-v1.5 | 768 | Default, good quality/speed
BAAI/bge-small-en-v1.5 | 384 | Faster, smaller
BAAI/bge-large-en-v1.5 | 1024 | Best quality, slower
sentence-transformers/all-MiniLM-L6-v2 | 384 | Fast, general purpose
sentence-transformers/all-mpnet-base-v2 | 768 | High quality
nomic-ai/nomic-embed-text-v1.5 | 768 | Long context (8192 tokens)

Ollama

Local Ollama server. Requires Ollama to be running.

embedder => {ollama, #{
    url => <<"http://localhost:11434">>,   %% Ollama API URL (default)
    model => <<"nomic-embed-text">>,       %% Model name (default, 768 dims)
    timeout => 30000                       %% Timeout in ms (default)
}}

Setup:

ollama pull nomic-embed-text

Supported Models:

Model | Dimensions | Notes
nomic-embed-text | 768 | Default, general purpose
mxbai-embed-large | 1024 | High quality
all-minilm | 384 | Fast
snowflake-arctic-embed | 1024 | Multilingual

OpenAI

OpenAI Embeddings API. Requires an API key.

embedder => {openai, #{
    api_key => <<"sk-...">>,               %% API key (or set OPENAI_API_KEY env var)
    model => <<"text-embedding-3-small">>, %% Model name (default, 1536 dims)
    timeout => 30000                       %% Timeout in ms (default)
}}

Supported Models:

Model | Dimensions | Notes
text-embedding-3-small | 1536 | Default, fast and cheap
text-embedding-3-large | 3072 | Higher quality
text-embedding-ada-002 | 1536 | Legacy model

FastEmbed

Lightweight ONNX-based embeddings. Faster than sentence-transformers for many models.

embedder => {fastembed, #{
    python => "python3",
    model => "BAAI/bge-small-en-v1.5",
    timeout => 120000
}}

Setup:

pip install fastembed

Provider Chain

Try providers in order until one succeeds:

embedder => [
    {openai, #{api_key => <<"sk-...">>}},
    {ollama, #{url => <<"http://localhost:11434">>}},
    {local, #{}}
]

Advanced Embeddings

SPLADE (Sparse Embeddings)

Neural sparse embeddings with term expansion for hybrid search.

{ok, State} = barrel_embed:init(#{
    embedder => {splade, #{
        model => "prithivida/Splade_PP_en_v1"
    }}
}).

{ok, SparseVec} = barrel_embed_splade:embed_sparse(<<"query text">>, Config).
%% => #{indices => [1, 42, 156], values => [0.5, 0.3, 0.8]}

Setup:

pip install transformers torch

ColBERT (Late Interaction)

Multi-vector embeddings for fine-grained token-level matching.

{ok, State} = barrel_embed:init(#{
    embedder => {colbert, #{
        model => "colbert-ir/colbertv2.0"
    }}
}).

{ok, MultiVec} = barrel_embed_colbert:embed_multi(<<"query text">>, Config).
%% => [[0.1, 0.2, ...], [0.3, 0.4, ...], ...]

Score = barrel_embed_colbert:maxsim_score(QueryVecs, DocVecs).

CLIP (Image Embeddings)

Cross-modal embeddings for image-text search.

{ok, State} = barrel_embed:init(#{
    embedder => {clip, #{
        model => "openai/clip-vit-base-patch32"
    }}
}).

{ok, TextVec} = barrel_embed_clip:embed(<<"a photo of a cat">>, Config).
{ok, ImageVec} = barrel_embed_clip:embed_image(Base64Image, Config).

Setup:

pip install transformers torch pillow

Model | Dimensions | Notes
openai/clip-vit-base-patch32 | 512 | Default, fast
openai/clip-vit-base-patch16 | 512 | Higher quality
openai/clip-vit-large-patch14 | 768 | Best quality

Reranking

Cross-encoder reranking for improved search relevance using the optional barrel_rerank package.

Installation

Add barrel_rerank to your dependencies:

%% rebar.config
{deps, [
    {barrel_vectordb, "2.1.1"},
    {barrel_rerank, "1.0.0"}
]}.

Usage

%% Start the reranker
{ok, Reranker} = barrel_rerank:start_link(#{
    model => "cross-encoder/ms-marco-MiniLM-L-6-v2"
}).

%% Two-stage retrieval
{ok, Candidates} = barrel_vectordb:search(Store, Query, #{k => 100}).

Docs = [maps:get(text, C) || C <- Candidates],
{ok, Ranked} = barrel_rerank:rerank(Reranker, Query, Docs).
%% => [{0, 0.95}, {2, 0.82}, {1, 0.45}, ...]

%% Stop when done
ok = barrel_rerank:stop(Reranker).

Supported Reranker Models:

Model | Notes
cross-encoder/ms-marco-MiniLM-L-6-v2 | Default, fast
cross-encoder/ms-marco-MiniLM-L-12-v2 | Better quality
BAAI/bge-reranker-base | Good quality
BAAI/bge-reranker-large | Best quality

BM25 Sparse Retrieval

Pure Erlang BM25 implementation for lexical search:

Index = barrel_vectordb_bm25:new().
Index1 = barrel_vectordb_bm25:add(Index, <<"doc1">>, <<"The quick brown fox">>).
Index2 = barrel_vectordb_bm25:add(Index1, <<"doc2">>, <<"The lazy dog">>).

Results = barrel_vectordb_bm25:search(Index2, <<"quick fox">>, 10).
%% => [{<<"doc1">>, 2.45}, ...]

Note

BM25 index is in-memory and not persisted. Rebuild from documents on startup.

barrel_vectordb Features

Additional features and advanced functionality for barrel_vectordb.

DiskANN Index

DiskANN is an SSD-optimized vector index based on the Vamana graph algorithm. It provides efficient billion-scale vector search with sub-linear memory usage.

Key Features

Two-pass Vamana construction - Builds a high-quality graph with alpha-RNG pruning

FreshVamana streaming updates - Insert and delete without full rebuild

Hot layer - Sub-millisecond writes absorbed in memory, compacted to disk

Lazy graph loading - O(1) startup time with on-demand node loading

PQ compression - Product quantization for reduced memory footprint

4KB sector-aligned I/O - Optimized for SSD performance

Usage

%% Create a new DiskANN index
{ok, Index} = barrel_vectordb_diskann:new(#{
    dimension => 768,
    r => 64,              %% Max out-degree
    l_build => 100,       %% Build search width
    l_search => 100,      %% Query search width
    alpha => 1.2,         %% Pruning factor (>1 for long-range edges)
    distance_fn => cosine %% cosine or euclidean
}).

%% Build from a list of vectors
Vectors = [{<<"id1">>, Vector1}, {<<"id2">>, Vector2}, ...],
{ok, Index1} = barrel_vectordb_diskann:build(Options, Vectors).

%% Insert a single vector
{ok, Index2} = barrel_vectordb_diskann:insert(Index1, <<"id3">>, Vector3).

%% Batch insert (more efficient)
{ok, Index3} = barrel_vectordb_diskann:insert_batch(Index2, [
    {<<"id4">>, Vector4},
    {<<"id5">>, Vector5}
], #{}).

%% Search for K nearest neighbors
Results = barrel_vectordb_diskann:search(Index3, QueryVector, 10).
%% => [{<<"id1">>, 0.95}, {<<"id2">>, 0.89}, ...]

%% Search with options
Results = barrel_vectordb_diskann:search(Index3, QueryVector, 10, #{
    l_search => 200  %% Higher = better recall, slower
}).

%% Delete a vector (lazy delete)
{ok, Index4} = barrel_vectordb_diskann:delete(Index3, <<"id1">>).

%% Check if consolidation needed (deleted > 10% of active)
case barrel_vectordb_diskann:needs_consolidation(Index4) of
    true -> {ok, Index5} = barrel_vectordb_diskann:consolidate_deletes(Index4);
    false -> Index5 = Index4
end.

%% Get index info
Info = barrel_vectordb_diskann:info(Index5).
%% => #{size => 4, dimension => 768, r => 64, ...}

Disk Mode

For large indexes, use disk mode for persistence and lazy loading:

%% Create with disk storage
{ok, Index} = barrel_vectordb_diskann:new(#{
    dimension => 768,
    storage_mode => disk,
    base_path => <<"/path/to/index">>
}).

%% Build and persist
{ok, Index1} = barrel_vectordb_diskann:build(Options, Vectors),
ok = barrel_vectordb_diskann:sync(Index1).

%% Open existing index (O(1) startup)
{ok, Index2} = barrel_vectordb_diskann:open(<<"/path/to/index">>).

%% Close when done
ok = barrel_vectordb_diskann:close(Index2).

Hot Layer

Enable the hot layer for sub-millisecond write latency:

{ok, Index} = barrel_vectordb_diskann:new(#{
    dimension => 768,
    storage_mode => disk,
    base_path => <<"/path/to/index">>,
    hot_enabled => true,
    hot_max_size => 10000,           %% Max vectors in hot layer
    hot_compaction_threshold => 0.8  %% Compact at 80% capacity
}).

%% Inserts go to hot layer first (sub-ms latency)
{ok, Index1} = barrel_vectordb_diskann:insert(Index, <<"id">>, Vector).

%% Search combines hot layer + disk results
Results = barrel_vectordb_diskann:search(Index1, Query, 10).

%% Manual compaction (hot layer -> disk)
{ok, Index2} = barrel_vectordb_diskann:compact(Index1).

Parameters

Parameter | Default | Description
dimension | required | Vector dimension
r | 64 | Max out-degree (connections per node)
l_build | 100 | Search width during construction
l_search | 100 | Default search width for queries
alpha | 1.2 | Pruning factor (>1 keeps long-range edges)
distance_fn | cosine | Distance function: cosine or euclidean
storage_mode | memory | memory or disk
use_pq | false | Enable product quantization

File Layout (Disk Mode)

/path/to/index/
  diskann.meta     # Erlang term metadata
  diskann.graph    # Vamana graph (4KB aligned nodes)
  diskann.vectors  # Full float32 vectors (4KB aligned)
  diskann.pq       # PQ codebooks + codes
  diskann.idmap    # ID to offset mapping

Document Backend (docstore)

By default a store keeps each vector's text and metadata in its own RocksDB column families, written in the same atomic batch as the vector. Set docstore to move them somewhere else, so documents and vectors can share one source of truth. Vectors always stay local.

barrel_vectordb_docdb_backend stores them as documents in a barrel_docdb database:

{ok, _} = barrel_vectordb:start_link(#{
    name => docs,
    dimension => 768,
    docstore => {barrel_vectordb_docdb_backend, #{db => <<"docs">>}}
}).

Options: db (the docdb database name, default the store name) and docdb_opts (passed to barrel_docdb:create_db/2). Metadata round-trips through term_to_binary, so atom keys and nested terms survive exactly.

The backend is optional and resolved at runtime from this config, so barrel_vectordb does not depend on barrel_docdb. Add barrel_docdb to your own application to use it. Writes are vector-first, doc-second: the vector batch commits, then text and metadata are written.

Implement barrel_vectordb_docstore to route them anywhere else.

Benchmarks

barrel_vectordb includes a comprehensive benchmark suite for measuring performance.

Quick Start

# Run all benchmarks
./scripts/run_benchmarks.sh

# Quick run (fewer iterations)
./scripts/run_benchmarks.sh --quick

# Full benchmark suite
./scripts/run_benchmarks.sh --full

# Export results
./scripts/run_benchmarks.sh --json
./scripts/run_benchmarks.sh --csv
./scripts/run_benchmarks.sh --all-formats

# Compare with baseline
./scripts/run_benchmarks.sh --compare baseline.json

Programmatic Usage

%% Run all benchmarks
rebar3 as bench shell
barrel_vectordb_bench:run_all().

%% Run with options
barrel_vectordb_bench:run_all(#{
    iterations => 100,
    warmup_iterations => 10,
    dimension => 128,
    output_format => json,  %% console | json | csv | all
    output_file => "bench_results"
}).

%% Run a specific benchmark
{ok, Result} = barrel_vectordb_bench:run(insert_single).
{ok, Result} = barrel_vectordb_bench:run(search_k10, #{iterations => 500}).

Available Benchmarks

Benchmark | Description
insert_single | Single vector insert
insert_batch_10 | Batch insert 10 vectors
insert_batch_100 | Batch insert 100 vectors
insert_batch_1000 | Batch insert 1000 vectors
search_k1 | Search for top 1 result
search_k10 | Search for top 10 results
search_k50 | Search for top 50 results
search_filtered | Search with metadata filter
index_build_1k | Build index with 1K vectors
index_build_10k | Build index with 10K vectors
get_single | Get single document by ID
delete_single | Delete single document
concurrent_writers | Concurrent write operations

Backend Comparison

Compare HNSW, FAISS, and DiskANN performance:

# Quick comparison
./scripts/run_backend_bench.sh --quick

# Default comparison
./scripts/run_backend_bench.sh

# Full benchmark suite
./scripts/run_backend_bench.sh --full

Or programmatically:

rebar3 as bench_faiss shell
barrel_vectordb_backend_bench:run_all().

Backend Overview

Backend | Memory Usage | Build Time | Search Latency | Best For
HNSW | High (~4KB/vector) | Fast | ~0.25ms | Small-medium datasets, low latency
FAISS | Medium | Fast | Very fast | GPU acceleration, batch queries
DiskANN | Low (~530B/vector in disk mode) | Slower | ~1ms | Large-scale datasets, memory-constrained

DiskANN Backend

DiskANN is ideal for billion-scale vector search where memory is limited:

Memory mode: Vectors in RAM, graph + PQ codes in memory

Disk mode: Vectors on SSD with LRU cache, ~7.5x memory reduction vs HNSW

See the DiskANN Index section above for detailed API documentation.

Benchmark Output

Results include:

ops_per_sec - Operations per second (throughput)

mean_ms - Mean latency in milliseconds

p50_ms - Median latency

p95_ms - 95th percentile latency

p99_ms - 99th percentile latency

min_ms / max_ms - Min/max latency

Example JSON output:

{
  "benchmark": "search_k10",
  "ops_per_sec": 12500,
  "mean_ms": 0.08,
  "p50_ms": 0.07,
  "p95_ms": 0.12,
  "p99_ms": 0.18,
  "iterations": 100
}

Comparing Results

# Run baseline
./scripts/run_benchmarks.sh --json
mv bench_results.json baseline.json

# Make changes, then compare
./scripts/run_benchmarks.sh --compare baseline.json

The comparison shows percentage differences for each metric, highlighting regressions and improvements.

Disk-Native BM25 Backend

barrel_vectordb provides a disk-native BM25 backend optimized for large-scale text search with predictable memory usage and fast query performance.

Overview

The bm25_disk backend stores inverted indexes on SSD with Block-Max MaxScore for early termination, similar to how production search engines like Elasticsearch and Lucene operate.

Key Features

Disk-native storage - Postings on SSD, only metadata in RAM

Block-Max MaxScore - Early termination skips 95%+ of postings

Hot layer - Sub-millisecond writes, background compaction

O(1) startup - Lazy loading, no full index load

Hybrid search - Combine BM25 + vector search with RRF or linear fusion

When to Use

Use Case | Recommended Backend
Small corpus (<100K docs) | memory - simpler, lower latency
Large corpus (>100K docs) | disk - predictable memory
Memory-constrained environments | disk
Write-heavy workloads | disk with large hot layer
Hybrid search (BM25 + vectors) | Either - both support hybrid

Memory Usage

Component | Size | Notes
Lexicon | ~40 bytes/term | Term → integer ID in RocksDB
Block-max index | ~50 bytes/block | mmap'd, OS cached
Doc stats | ~8 bytes/doc | In ETS
Hot layer | configurable | Default 50K docs max

Example sizing for 1M documents with 500K unique terms:

Memory backend: ~500 MB - 1 GB

Disk backend: ~50 MB + OS page cache

Configuration

Standalone Store

%% Create store with disk BM25
{ok, _} = barrel_vectordb:start_link(#{
    name => my_store,
    path => "/data/my_store",
    dimensions => 768,
    bm25_backend => disk,
    bm25_disk => #{
        base_path => "/data/my_store/bm25",
        hot_max_size => 50000,
        hot_compaction_threshold => 0.8,
        k1 => 1.2,
        b => 0.75
    }
}).

Configuration Options

Option | Default | Description
base_path | {store_path}/bm25 | Directory for BM25 files
hot_max_size | 50000 | Max docs in hot layer before compaction
hot_compaction_threshold | 0.8 | Trigger compaction at this % of max
k1 | 1.2 | BM25 term frequency saturation
b | 0.75 | BM25 document length normalization
block_size | 128 | Documents per posting block

BM25 Parameters

The BM25 scoring formula is:

score = sum(IDF(t) * TF(t,d) * (k1 + 1) / (TF(t,d) + k1 * (1 - b + b * |d| / avgdl)))

k1 (default: 1.2) - Controls term frequency saturation. Higher values give more weight to repeated terms.

b (default: 0.75) - Controls document length normalization. 0 = no normalization, 1 = full normalization.

API Reference

Adding Documents

Documents are automatically indexed in BM25 when added to the store:

%% Add single document - automatically indexed in BM25
ok = barrel_vectordb:add(Store, <<"doc1">>, <<"Hello world">>, #{}).

%% Add with metadata
ok = barrel_vectordb:add(Store, <<"doc2">>, <<"Erlang is great">>,
                         #{type => article, author => <<"joe">>}).

%% Batch add (more efficient)
{ok, _} = barrel_vectordb:add_batch(Store, [
    {<<"doc3">>, <<"First document">>, #{}},
    {<<"doc4">>, <<"Second document">>, #{}}
]).

BM25 Search

%% Basic BM25 search
{ok, Results} = barrel_vectordb:search_bm25(Store, <<"erlang programming">>, #{k => 10}).
%% Returns: [{<<"doc2">>, 2.45}, {<<"doc1">>, 1.82}, ...]

%% With more results
{ok, Results} = barrel_vectordb:search_bm25(Store, <<"erlang">>, #{k => 50}).

Hybrid Search (BM25 + Vector)

Combine keyword matching with semantic similarity:

%% Combined search with RRF fusion (recommended)
{ok, Results} = barrel_vectordb:search_hybrid(Store, <<"erlang concurrency">>, #{
    k => 10,
    bm25_weight => 0.5,
    vector_weight => 0.5,
    fusion => rrf
}).

%% Linear fusion (simpler, score-based)
{ok, Results} = barrel_vectordb:search_hybrid(Store, <<"erlang">>, #{
    k => 10,
    bm25_weight => 0.3,
    vector_weight => 0.7,
    fusion => linear
}).

Fusion Algorithms:

Algorithm | Description | Best For
rrf | Reciprocal Rank Fusion with k=60 | General use, rank-based
linear | Weighted sum of normalized scores | When score magnitude matters

Index Management

%% Force compaction (hot layer -> disk)
ok = barrel_vectordb_server:bm25_compact(Store).

%% Get index statistics
{ok, Stats} = barrel_vectordb_server:bm25_info(Store).
%% Returns: #{
%%     backend => disk,
%%     total_docs => 100000,
%%     vocab_size => 50000,
%%     hot_docs => 1234,
%%     disk_docs => 98766,
%%     avgdl => 150.5
%% }

Architecture

File Layout

/data/my_store/bm25/
├── bm25.meta           # Header + global stats (4KB aligned)
├── bm25.postings       # Compressed posting blocks (4KB aligned)
├── bm25.blockmax       # Block-max index (mmap'd)
└── bm25.ids/           # RocksDB: term/doc ID mapping
    ├── terms_fwd       # term string → integer ID
    ├── terms_rev       # integer ID → term string
    ├── docs_fwd        # doc string ID → integer ID
    └── docs_rev        # integer ID → doc string ID

Block-Max MaxScore Algorithm

The search algorithm efficiently skips blocks that cannot contribute to top-k:

1. Initialize
   - Load block-max lists for query terms
   - Set threshold = 0

2. For each document position:
   a. Find pivot: smallest doc_id where sum(max_impacts) > threshold
   b. For each term's block at pivot:
      - If block.max_impact < threshold - accumulated: SKIP block
      - Otherwise: decompress and score
   c. If score > threshold:
      - Add to result heap
      - Update threshold = heap.min_score

3. Return top-K from heap

This typically skips 95%+ of postings for selective queries.

Hot Layer Architecture

New documents go to an in-memory hot layer for fast writes:

Write Request
     │
     ▼
┌─────────────┐
│  Hot Layer  │ ← In-memory maps
│  (fast)     │   Sub-ms writes
└──────┬──────┘
       │ Compaction (at 80% capacity)
       ▼
┌─────────────┐
│ Disk Layer  │ ← Block-max postings
│ (large)     │   mmap'd block-max index
└─────────────┘
       │
       ▼
   Search combines both layers

Varint Encoding

Postings use delta-encoded varints for compact storage:

Original doc IDs:  [100, 105, 108, 150]
Delta encoded:     [100,   5,   3,  42]
Varint bytes:      [0x64, 0x05, 0x03, 0x2A]  (4 bytes vs 16 bytes)

Performance Tuning

Memory vs Latency Tradeoffs

Setting | Effect
Increase hot_max_size | More RAM, fewer compactions, faster writes
Decrease hot_max_size | Less RAM, more frequent compactions
Decrease block_size | Finer skip granularity, more overhead
Larger OS page cache | More hot blocks in RAM, lower disk reads

Recommended Settings

Write-heavy workload:

#{
    hot_max_size => 100000,
    hot_compaction_threshold => 0.9
}

Read-heavy workload:

#{
    hot_max_size => 10000,
    hot_compaction_threshold => 0.5
}

Memory-constrained:

#{
    hot_max_size => 5000
    %% Rely on disk + OS page cache
}

High-throughput hybrid search:

#{
    hot_max_size => 50000,
    %% Ensure embedder is configured for vector search
}

Comparison with Memory Backend

Feature | Memory Backend | Disk Backend
Storage | In-memory maps | Disk + hot layer
Startup | Load all data | O(1) lazy load
Memory | O(corpus size) | O(vocab + hot)
Write latency | ~microseconds | ~milliseconds
Search latency | ~milliseconds | ~milliseconds
Max corpus | Limited by RAM | Limited by SSD
Persistence | No | Yes

Migration Guide

From Memory Backend

%% 1. Stop writes to the store
%% 2. Update config to use disk backend
NewConfig = OldConfig#{bm25_backend => disk},

%% 3. Restart store - documents need to be re-indexed
barrel_vectordb:stop(Store),
{ok, _} = barrel_vectordb:start_link(NewConfig).

%% 4. Re-index existing documents if needed
%% (Documents added after restart will be auto-indexed)

From External Search Engine

%% Export {Id, Text, Meta} tuples from your existing search engine, then
%% index them the same way as any other document:
lists:foreach(fun({Id, Text, Meta}) ->
    barrel_vectordb:add(Store, Id, Text, Meta)
end, Documents).

Troubleshooting

High Memory Usage

If memory usage is higher than expected:

Check hot layer size: barrel_vectordb_server:bm25_info(Store)

Reduce hot_max_size in config

Lower hot_compaction_threshold for more frequent compaction

Slow Search

If BM25 searches are slow:

Ensure OS has sufficient page cache for block-max index

Check if hot layer is very large (may need compaction)

For multi-term queries, ensure block-max index is loaded

Disk Space

The disk backend uses approximately:

~50 bytes per posting (term-doc pair) after compression

~20 bytes per block-max entry

RocksDB overhead for ID mapping

For 1M documents with average 100 terms each:

Postings: ~5 GB

Block-max: ~100 MB

ID mapping: ~50 MB

Testing

The BM25 implementation includes tests for formula correctness and relevance quality.

Running BM25 Tests

# Formula correctness tests (11 tests)
rebar3 eunit --module=barrel_vectordb_bm25_formula_tests

# Corpus-based relevance tests (10 tests)
rebar3 eunit --module=barrel_vectordb_bm25_corpus_tests

# Both modules
rebar3 eunit --module=barrel_vectordb_bm25_formula_tests,barrel_vectordb_bm25_corpus_tests

# All BM25 tests (includes memory backend tests)
rebar3 eunit --module=barrel_vectordb_bm25_tests,barrel_vectordb_bm25_disk_tests,barrel_vectordb_bm25_formula_tests,barrel_vectordb_bm25_corpus_tests

Formula Correctness Tests

Located in test/barrel_vectordb_bm25_formula_tests.erl, these tests verify the BM25 scoring formula with hand-calculated expected values:

Test | Description
single_term_single_doc | Verifies basic BM25 score calculation
multi_term_score | Scores should sum across query terms
tf_impact | Higher TF should increase score
idf_impact | Rare terms should have higher IDF contribution
doc_length_normalization | Shorter docs favored with equal TF
k1_parameter | k1=0 saturates TF immediately, k1=high allows more TF effect
b_parameter | b=0 no length normalization, b=1 full normalization
tf_saturation | Score(TF=10) < 2 × Score(TF=5) due to diminishing returns
edge_cases | Empty queries, missing terms, single doc corpus
hand_calculated_example | Full worked example from documentation
backend_consistency | Memory and disk backends produce identical scores

Corpus-Based Relevance Tests

Located in test/barrel_vectordb_bm25_corpus_tests.erl, these tests evaluate search quality using IR metrics:

Test Corpus: 100 documents across 5 topic clusters:

Erlang/OTP (docs 1-20)

Python/Data Science (docs 21-40)

Java/Enterprise (docs 41-60)

Distributed Systems (docs 61-80)

General Programming (docs 81-100)

Test Queries: 20 queries with relevance judgments covering:

Single-term topic queries

Multi-term queries

Cross-topic queries

Rare term queries (high IDF)

Common term queries (low IDF)

Metrics Evaluated:

Test | Description
precision_at_5 | P@5 averaged across all queries
precision_at_10 | P@10 averaged across all queries
recall_at_10 | R@10 for highly relevant documents
ndcg_at_10 | Normalized DCG with graded relevance
relevant_in_top_k | At least one relevant doc in top 5 for topic queries
ranking_order | Results sorted by descending score
cross_topic_queries | Queries matching multiple topic clusters
rare_term_queries | High IDF queries return correct results
memory_disk_consistency | Both backends produce similar results

Test Data Files

File | Description
test/data/bm25_test_corpus.terms | 100 documents in Erlang term format
test/data/bm25_test_queries.terms | 20 queries with relevance judgments

Adding Custom Test Documents

The corpus file uses Erlang term format:

%% test/data/bm25_test_corpus.terms
[
    {<<"doc1">>, <<"Your document text here">>},
    {<<"doc2">>, <<"Another document">>},
    ...
].

Query judgments specify relevant and partially relevant documents:

%% test/data/bm25_test_queries.terms
[
    {<<"query text">>, #{
        relevant => [1, 2, 3],      %% Highly relevant doc numbers
        partial => [10, 15]         %% Partially relevant
    }},
    ...
].

TurboQuant Vector Quantization

TurboQuant provides efficient vector compression for embedding storage with no training required. Part of barrel_vectordb, it's ideal for reducing memory usage in large-scale vector search applications.

Overview

TurboQuant is a data-oblivious 3-bit vector quantization algorithm based on Google Research's PolarQuant/QJL technique. Unlike Product Quantization (PQ), it requires no training phase and delivers deterministic, reproducible results.

Key benefits:

No training required - Works immediately on any data

Deterministic - Same seed produces same results

~8x compression - 768-dim vectors: 3KB to ~388 bytes

1-3% recall loss - Minimal accuracy impact vs float32

SIMD-accelerated - AVX2/NEON optimized distance computation

Quick Start

Basic Encode/Decode

%% Create quantizer (no training needed)
{ok, TQ} = barrel_vectordb_turboquant:new(#{
    dimension => 768,
    bits => 3
}).

%% Encode a vector
Vector = lists:seq(1, 768),  %% Your embedding
Code = barrel_vectordb_turboquant:encode(TQ, Vector).
%% Code is ~388 bytes vs 3072 bytes (768 * 4)

%% Decode back to approximate vector
Decoded = barrel_vectordb_turboquant:decode(TQ, Code).

Distance Computation

%% Precompute lookup tables for query (do once per query)
Query = [0.1, 0.2, ...],  %% 768-dim query vector
Tables = barrel_vectordb_turboquant:precompute_tables(TQ, Query).

%% Fast distance computation (SIMD-accelerated NIF)
Distance = barrel_vectordb_turboquant:distance_nif(Tables, Code).

%% Batch distance for multiple codes
Codes = [Code1, Code2, Code3],
Distances = barrel_vectordb_turboquant:batch_distance_nif(Tables, Codes).

Configuration Options

{ok, TQ} = barrel_vectordb_turboquant:new(#{
    dimension => 768,           %% Required, must be even
    bits => 3,                  %% 2-4, default: 3
    seed => 42,                 %% Random seed, default: 42
    qjl_iterations => 5,        %% Error correction iterations, default: 5
    qjl_learning_rate => 0.1    %% Gradient step size, default: 0.1
}).

Option | Type | Default | Description
dimension | integer | required | Vector dimension (must be even)
bits | 2-4 | 3 | Bits per polar angle (higher = more accurate, larger)
seed | integer | 42 | Random seed for rotation matrix
qjl_iterations | integer | 5 | QJL error correction iterations
qjl_learning_rate | float | 0.1 | Gradient descent step size

API Reference

new/1

Create a new TurboQuant configuration.

-spec new(map()) -> {ok, tq_config()} | {error, term()}.

encode/2

Encode a vector to compact binary representation.

-spec encode(tq_config(), [float()]) -> tq_code().

decode/2

Decode a TurboQuant code back to approximate vector.

-spec decode(tq_config(), tq_code()) -> [float()].

precompute_tables/2

Precompute distance lookup tables for a query vector. Call once per query, then use for many distance computations.

-spec precompute_tables(tq_config(), [float()]) -> distance_tables().

distance/2

Compute asymmetric distance using precomputed tables (pure Erlang).

-spec distance(distance_tables(), tq_code()) -> float().

distance_nif/2

Compute ADC distance using SIMD-accelerated NIF. Use this for production workloads.

-spec distance_nif(distance_tables(), tq_code()) -> float().

batch_distance_nif/2

Compute ADC distance for multiple codes. Amortizes NIF call overhead for batch operations.

-spec batch_distance_nif(distance_tables(), [tq_code()]) -> [float()].

batch_encode/2

Encode multiple vectors.

-spec batch_encode(tq_config(), [[float()]]) -> [tq_code()].

info/1

Get configuration info including compression ratio.

-spec info(tq_config()) -> map().

Returns:

#{
    bits => 3,
    qjl_bits => 1,
    dimension => 768,
    rotation_seed => 42,
    qjl_iterations => 5,
    qjl_learning_rate => 0.1,
    bytes_per_vector => 388,
    compression_ratio => 7.92,
    training_required => false
}

Subspace-TurboQuant

For large dimensions (1024+), Subspace-TurboQuant improves performance by splitting vectors into independent subspaces. This reduces rotation matrix memory from O(D^2) to O(D^2/M).

When to Use

Dimensions > 768

Memory-constrained environments

When encode latency matters

API

%% Create with auto-selected M
{ok, TQS} = barrel_vectordb_turboquant_subspace:new(#{
    dimension => 1536
}).

%% Or specify M explicitly
{ok, TQS} = barrel_vectordb_turboquant_subspace:new(#{
    dimension => 1536,
    m => 16  %% 16 subspaces of 96 dimensions each
}).

%% API is identical to TurboQuant
Code = barrel_vectordb_turboquant_subspace:encode(TQS, Vector),
Tables = barrel_vectordb_turboquant_subspace:precompute_tables(TQS, Query),
Distance = barrel_vectordb_turboquant_subspace:distance_nif(Tables, Code).

Auto M Selection

Dimension | M | Subdim
<= 128 | 1 | D
<= 256 | 2 | D/2
<= 512 | 4 | D/4
<= 1024 | 8 | D/8
<= 2048 | 16 | D/16
> 2048 | 32 | D/32

Memory Comparison

For D=768:

Variant | Rotation Memory | Encode Latency
TurboQuant | 4.7MB | ~3.6ms
Subspace (M=8) | 590KB | ~0.5ms

Performance

Compression Ratios

Dimension | Bits | Bytes | Compression
384 | 3 | 196 | 7.8x
768 | 3 | 388 | 7.9x
1536 | 3 | 772 | 8.0x
768 | 2 | 340 | 9.0x
768 | 4 | 436 | 7.0x

NIF Speedup

Distance computation performance (768-dim, 1000 vectors):

Method | Time | Speedup
distance/2 (Erlang) | ~50ms | 1x
distance_nif/2 (C) | ~2ms | 25x
batch_distance_nif/2 | ~1.5ms | 33x

Recall vs Compression

At 3 bits with default settings:

Dataset | Recall@10 (float32) | Recall@10 (TurboQuant) | Loss
SIFT1M | 98.2% | 96.1% | -2.1%
GloVe | 97.5% | 95.8% | -1.7%
OpenAI | 99.1% | 97.3% | -1.8%

Integration with HNSW

TurboQuant integrates with barrel_vectordb's HNSW index for compressed vector search:

%% Create HNSW index with TurboQuant compression
Index = barrel_vectordb_hnsw:new(#{
    dimension => 768,
    distance_fn => cosine,
    quantization => turboquant,
    tq_bits => 3
}).

%% Add vectors (automatically quantized)
Index1 = barrel_vectordb_hnsw:insert(Index, Id, Vector).

%% Search (uses ADC for distance computation)
Results = barrel_vectordb_hnsw:search(Index1, Query, 10).

When Quantization Happens

Index creation - TurboQuant config is initialized

Add vector - Vector is encoded and stored as compact code

Search - Query tables are precomputed once, ADC used for all distance computations

Reranking - Optional: decode top candidates for exact distance reranking

Best Practices

Reuse tables - Precompute tables once per query, reuse for all distance computations

Use NIF functions - Always prefer distance_nif/2 over distance/2 in production

Batch operations - Use batch_distance_nif/2 for multiple codes

Consider subspace - For D > 768, Subspace-TurboQuant offers better memory/latency tradeoffs

Tune bits - 3 bits is a good default; use 4 for higher accuracy, 2 for more compression

TurboQuant Internals

This document describes the internal implementation of TurboQuant vector quantization in barrel_vectordb.

Algorithm Overview

TurboQuant combines three techniques:

PolarQuant - Random rotation followed by polar coordinate conversion

QJL - Johnson-Lindenstrauss transform for error correction

ADC - Asymmetric Distance Computation for fast search

Vector ─> Rotate ─> Polar Coords ─> Quantize ─> QJL Signs ─> Binary Code
                                                                  │
Query ──> Rotate ─> Precompute Tables ────────────────────────────┼─> ADC Distance
                                                                  │
                                                        Compare signs for correction

Rotation Matrix

Generation

The rotation matrix is generated via Modified Gram-Schmidt orthogonalization of a random Gaussian matrix. This produces an orthogonal matrix Q where Q^T = Q^-1.

%% Generate D x D orthogonal rotation matrix
generate_rotation_matrix(Dim, Seed) ->
    rand:seed(exsss, {Seed, Seed + 1, Seed + 2}),
    RandomCols = [random_gaussian_vector(Dim) || _ <- lists:seq(1, Dim)],
    OrthogonalCols = modified_gram_schmidt(RandomCols),
    Rows = transpose(OrthogonalCols),
    << <<F:64/float-little>> || Row <- Rows, F <- Row >>.

Storage Format

Binary format: Row-major float64 (IEEE 754 double precision)

Size: D D 8 bytes (e.g., 768 768 8 = 4.7MB for D=768)

Access pattern: Sequential row access during rotation

Caching Strategy

The rotation matrix is generated once per TurboQuant config and stored in the config record. Regeneration uses the same seed for deterministic results.

Polar Coordinate Conversion

Cartesian to Polar

Vector components are paired and converted to polar form:

(x_i, y_i) -> (r_i, theta_i)
where:
  r = sqrt(x^2 + y^2)
  theta = atan2(y, x)

This yields D/2 radius-angle pairs from a D-dimensional vector.

Radius Quantization

Radii are quantized to 16-bit using a log scale for better dynamic range:

quantize_radius(R) ->
    LogR = math:log(R + 1.0),
    Scaled = LogR / math:log(11.0) * 65535,
    min(65535, max(0, round(Scaled))).

dequantize_radius(QuantizedR) ->
    LogR = QuantizedR / 65535.0 * math:log(11.0),
    math:exp(LogR) - 1.0.

The log scale maps typical embedding radii [0, 10] to the full 16-bit range.

Angle Quantization

Angles are quantized to N-bit buckets (N = bits parameter):

NumLevels = 2^bits  (e.g., 8 levels for 3-bit)
BucketSize = 2*pi / NumLevels
AngleIdx = floor((theta + pi) / BucketSize)

Angle bucket centers are precomputed for fast lookup during decoding:

compute_angle_levels(NumLevels) ->
    BucketSize = 2 * math:pi() / NumLevels,
    [(-math:pi() + BucketSize * (I + 0.5)) || I <- lists:seq(0, NumLevels - 1)].

QJL Error Correction

Sign Matrix Generation

The QJL matrix is a D x D random sign matrix where each element is +1 or -1:

generate_qjl_matrix(Dim, QJLDim, Seed) ->
    rand:seed(exsss, {Seed, Seed + 1, Seed + 2}),
    Signs = [rand:uniform(2) - 1 || _ <- lists:seq(1, Dim * QJLDim)],
    pack_bits_simple(Signs).  %% Store as bits: 1 = +1, 0 = -1

Storage: D^2 / 8 bytes (e.g., 73KB for D=768).

Sign Computation

During encoding, compute signs of the projection:

sign_i = sign(QJL_row_i . Vector)

These signs are stored with the code and used during decoding to refine the reconstruction.

Iterative Refinement

During decoding, the algorithm iteratively adjusts the reconstructed vector to match stored signs:

apply_qjl_correction(Vector, StoredSigns, QJLMatrix, QJLDim, Iterations, LR) ->
    %% For each iteration:
    %%   1. Compute current signs
    %%   2. Find mismatches with stored signs
    %%   3. Apply gradient toward sign-consistent solution

The gradient for a sign mismatch at row i is:

gradient += target_sign * QJL_row_i

Learning rate and iteration count control convergence:

Default: 5 iterations, LR = 0.1

Gradient is normalized relative to vector magnitude to prevent overshooting

Binary Code Format

TurboQuant Code Structure

+--------+------+-------+----------+----------+
| Header |  Radii       |  Angles  | QJL Signs|
+--------+--------------+----------+----------+
| 4B     | D/2 * 2B     | ceil(D/2 * bits/8) | ceil(D/8) |

Header format:

<<Version:8, Bits:8, Flags:16>>

Version: 1 for TurboQuant, 2 for Subspace-TurboQuant

Bits: Quantization bits (2-4)

Flags: Reserved

Subspace-TurboQuant Code Structure

+--------+----------------+----------------+-----+
| Header | Subspace1 Code | Subspace2 Code | ... |
+--------+----------------+----------------+-----+
| 4B     | Variable       | Variable       |     |

Header format:

<<Version:8, Bits:8, M:8, Flags:8>>

Each subspace code contains radii, angles, and QJL signs for that subspace (no individual headers).

Bit Packing

Angle indices are bit-packed for efficiency:

%% Pack list of N-bit integers into binary
pack_bits([Idx | Rest], Bits, Acc, Buffer, BufferBits) ->
    NewBuffer = (Buffer bsl Bits) bor Idx,
    NewBufferBits = BufferBits + Bits,
    case NewBufferBits >= 8 of
        true ->
            %% Extract complete bytes
            extract_bytes(Acc, NewBuffer, NewBufferBits);
        false ->
            pack_bits(Rest, Bits, Acc, NewBuffer, NewBufferBits)
    end.

ADC Distance Formula

Table Precomputation

For each pair (D/2 pairs total), precompute:

QRSq: Query radius squared = r_q^2

CosTerms: For each angle bucket i: 2 r_q cos(theta_q - angle_center_i)

Table size per pair: (1 + NumLevels) * 4 bytes

precompute_tables(Config, Query) ->
    RotatedQuery = apply_rotation(RotMat, Query),
    Tables = lists:map(
        fun(PairIdx) ->
            {QX, QY} = get_pair(RotatedQuery, PairIdx),
            QR = math:sqrt(QX*QX + QY*QY),
            QRSq = QR * QR,
            QTheta = math:atan2(QY, QX),
            CosTerms = [2.0 * QR * math:cos(QTheta - AngleCenter)
                        || AngleCenter <- Levels],
            {QRSq, CosTerms}
        end,
        lists:seq(0, NumPairs - 1)
    ),
    %% Pack as binary
    ...

Distance Computation

Per-pair distance using the polar distance formula:

d^2 = r_q^2 + r_d^2 - 2*r_q*r_d*cos(theta_diff)
    = QRSq + DR^2 - CosTerm[angle_idx] * DR

Total distance:

Distance = sqrt(sum over all pairs of (QRSq + DR^2 - CosTerm * DR))

SIMD Implementation

The NIF implements ADC with AVX2/NEON intrinsics:

// Pseudocode for SIMD ADC
float tq_adc_distance_simd(tables, code, bits, num_pairs) {
    __m256 sum_sq = _mm256_setzero_ps();

    for (int i = 0; i < num_pairs; i += 8) {
        // Load 8 radii and angle indices
        __m256 dr = load_and_dequantize_radii(code, i);
        __m256i angle_idx = load_angle_indices(code, i);

        // Gather QRSq and CosTerms from tables
        __m256 qrsq = gather_qrsq(tables, i);
        __m256 costerm = gather_costerm(tables, angle_idx, i);

        // Compute: QRSq + DR^2 - CosTerm * DR
        __m256 dr_sq = _mm256_mul_ps(dr, dr);
        __m256 contrib = _mm256_fmadd_ps(costerm, dr, dr_sq);
        contrib = _mm256_add_ps(qrsq, _mm256_sub_ps(dr_sq, contrib));

        sum_sq = _mm256_add_ps(sum_sq, contrib);
    }

    return sqrt(horizontal_sum(sum_sq));
}

Subspace Partitioning

Why Subspaces?

Standard TurboQuant has O(D^2) scaling due to the rotation matrix. For large D, this becomes prohibitive:

D | Rotation Memory | Encode Time
768 | 4.7MB | 3.6ms
1536 | 18.9MB | 14.4ms
3072 | 75.5MB | 57.6ms

Subspace Approach

Split D-dimensional vector into M subspaces of dimension D/M:

Vector = [v_1, ..., v_D]
       = [subvec_1, subvec_2, ..., subvec_M]

Each subspace has its own rotation matrix: M matrices of (D/M)^2 elements = D^2/M total.

split_subvectors(Vec, M, SubDim) ->
    split_subvectors(Vec, M, SubDim, []).

split_subvectors([], 0, _SubDim, Acc) ->
    lists:reverse(Acc);
split_subvectors(Vec, Remaining, SubDim, Acc) when Remaining > 0 ->
    {Subvec, Rest} = lists:split(SubDim, Vec),
    split_subvectors(Rest, Remaining - 1, SubDim, [Subvec | Acc]).

Distance Aggregation

Subspace distances are computed independently and summed:

Total_distance^2 = sum over subspaces of distance_subspace^2
Distance = sqrt(Total_distance^2)

This preserves the L2 distance property since:

||v - q||^2 = sum_i (v_i - q_i)^2
            = sum_subspaces sum_dims_in_subspace (v_i - q_i)^2

Trade-offs

Aspect | TurboQuant | Subspace (M=8)
Memory | D^2 * 8 | D^2/M * 8
Encode | O(D^2) | O(D^2/M)
Accuracy | Baseline | -1-2% recall
Parallelism | Serial | Per-subspace

The slight accuracy loss comes from quantizing rotation independently per subspace rather than globally.

Contributing

Guidelines for contributing to Barrel VectorDB.

Development Setup

Prerequisites

Erlang/OTP 26 or later

rebar3

Python 3.8+ (for embedding providers)

Clone and Build

git clone https://github.com/barrel-db/barrel.git
cd barrel/apps/barrel_vectordb
rebar3 compile

Run Tests

# Unit tests (no external dependencies)
rebar3 eunit

# All tests including integration
rebar3 ct

Testing

Unit Tests

Unit tests use mocking and don't require external dependencies:

rebar3 eunit

Integration Tests

Integration tests verify real embedding providers. Setup required backends first:

# Setup for local provider
pip install sentence-transformers

# Setup for Ollama provider
ollama serve &
ollama pull nomic-embed-text

# Setup for OpenAI provider
export OPENAI_API_KEY=sk-...

# Run integration tests
rebar3 eunit --module=barrel_vectordb_integration_tests

Tests automatically skip if their required backend is unavailable.

Benchmarks

# Run benchmark suite
rebar3 as bench compile && rebar3 as bench eunit --module=barrel_vectordb_bench

# Backend comparison
./scripts/run_backend_bench.sh --quick

Code Style

Follow standard Erlang conventions

Use edoc for documentation

Keep functions focused and small

Add tests for new functionality

Module Structure

-module(barrel_vectordb_example).

%% API
-export([public_function/1]).

%% Internal exports
-export_type([my_type/0]).

-type my_type() :: term().

%% @doc Public function documentation.
-spec public_function(Arg) -> Result when
    Arg :: term(),
    Result :: ok | {error, term()}.
public_function(Arg) ->
    internal_function(Arg).

%% Internal functions

internal_function(Arg) ->
    ok.

Pull Requests

Fork the repository

Create a feature branch

Make your changes

Add tests

Run the test suite

Submit a pull request

Commit Messages

Use clear, descriptive commit messages:

Add support for new embedding provider

- Add barrel_embed_newprovider.erl in apps/barrel_embed
- Update barrel_embed to support new provider
- Add unit tests

Architecture

Core Components

barrel_vectordb.erl: Main API module

barrel_vectordb_store.erl: Store process (gen_batch_server)

barrel_vectordb_hnsw.erl: HNSW index implementation

barrel_embed.erl (apps/barrel_embed): Embedding provider abstraction

Storage

RocksDB with column families

Vectors stored with 8-bit quantization

HNSW index checkpoints for fast restart

Reporting Issues

Use the GitHub issue tracker:

Report Issues

Include:

Erlang/OTP version

barrel_vectordb version

Steps to reproduce

Expected vs actual behavior

Relevant logs

License

Contributions are licensed under Apache-2.0.

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[2.1.1] - 2026-07-10

Fixed

barrel_vectordb_docdb_backend:init/2 raised badarg on every start. It called atom_to_binary/2 on a store name the store had already normalised to a binary, and maps:get/3 evaluates its default eagerly, so it crashed even when db was supplied. The docstore seam was unusable.

Changed

The barrel_vectordb_docstore init/2 callback accepts an atom or a binary name, which is what the store passes.

Added

docstore is documented in docs/features.md, and the docdb-backed backend has tests.

[2.1.0] - 2026-07-08

Coordinated umbrella release. See the umbrella CHANGELOG.

[2.0.0] - 2026-06-14

Removed

Clustering: removed the Ra/Raft mesh, sharding, scatter-gather, and the cluster API (start_cluster, cluster_join, cluster_status, create_collection, cluster_add, cluster_search, and related functions).

HTTP API: removed the cowboy-based HTTP server, routes, and handlers.

Multi-tenant gateway: removed the gateway, API keys, quotas, and rate limiting.

Dropped the ra and cowboy dependencies.

barrel_vectordb is now an embedded-only library. The cluster and HTTP code is preserved on the backup/cluster-http-api branch.

[1.5.0] - 2026-02-03

Added

BM25 Disk Backend: RocksDB-based persistent BM25 index with Block-Max MaxScore algorithm

Disk-native storage using RocksDB column families for term posting lists

Block-Max indexing for faster query evaluation with early termination

MaxScore optimization that skips low-scoring documents

Configurable block size (default: 128 documents per block)

Full persistence with automatic recovery on restart

New module: barrel_vectordb_bm25_disk

BM25 Cluster Integration: Distributed BM25 search with scatter-gather

Scatter queries across shards, gather and merge results

IDF synchronization across cluster for consistent scoring

Global document statistics for accurate term weighting

Works with both memory and disk backends

BM25 HTTP Endpoints: REST API for BM25 and hybrid search

POST /vectordb/collections/:collection/search/bm25 - Keyword search

POST /vectordb/collections/:collection/search/hybrid - Combined BM25 + vector search

Fusion algorithms: RRF (Reciprocal Rank Fusion) and linear combination

Configurable weights for BM25 and vector components

BM25 Formula Correctness Tests: Verification suite with hand-calculated expected scores

11 formula tests covering TF/IDF impact, parameter effects, edge cases

100-document test corpus across 5 topic clusters

20 test queries with relevance judgments

IR evaluation metrics: Precision@K, Recall@K, nDCG@10

BM25 Performance Benchmarks: Benchmark suite for BM25 backends

Memory vs disk backend comparison

Index build, search, and update performance metrics

Fixed

Fix hybrid_search benchmark to handle missing embedder gracefully

[1.4.0] - 2026-01-18

Added

Multi-Tenant HTTP Gateway: REST API with complete multi-tenancy support

Tenant isolation via API key authentication (X-Api-Key header)

Transparent collection name prefixing ({hash}_{tenant}_{collection})

Token bucket rate limiting per tenant (configurable RPM)

Quota enforcement for vectors, collections, and storage per tenant

Backend abstraction supporting both standalone and clustered deployments

Admin endpoints for tenant management, API key rotation, and usage monitoring

New modules: barrel_vectordb_gateway, barrel_vectordb_gateway_keys, barrel_vectordb_gateway_quotas, barrel_vectordb_gateway_rate, barrel_vectordb_gateway_stores, barrel_vectordb_system_db

Docker support with docker-compose.gateway.yml and environment variables

See Gateway Documentation for full details

[1.3.1] - 2026-01-10

Fixed

Fix cluster collection creation timeout with retry logic:

Increase DEFAULT_TIMEOUT from 5000ms to 10000ms

Add LONG_TIMEOUT (30000ms) for create_collection operations

Add retry logic with exponential backoff for node registration

Nodes now retry up to 5 times when registering in the Ra state machine

Fixes bug where nodes join Ra Raft cluster but fail to register in application-level state machine

[1.3.0] - 2026-01-09

Changed

Remove jsx dependency: Replace jsx library with Erlang's built-in json module (OTP 27+)

jsx:encode/1 → iolist_to_binary(json:encode/1)

jsx:decode/2 → json:decode/1

Fixed

Fix 49 dialyzer warnings across cluster and embedding modules:

Add aten, gen_batch_server to plt_extra_apps

Fix unmatched return values in Ra, cluster events, discovery, health, mesh modules

Fix typo: barrel_vectordb_embedder → barrel_vectordb_embed in scatter module

Fix key name: dimension → dimensions in shard_manager

Fix unreachable patterns in http_handlers and reshard modules

Replace lager:error with logger:error in embed_local

Update create_collection spec to match Ra state machine return type

Added

Comprehensive test coverage for scatter module (19 new tests)

Tests for search/4, search_vector/3, search_local_shard/3

Tests for embedder configuration, local/remote shards, RPC handling

Tests for result gathering, deduplication, and score ordering

[1.2.5] - 2026-01-05

Fixed

Fix dimension config propagation: pass dimensions key to embed provider init in barrel_vectordb_store

Add error handling for empty embeddings in barrel_vectordb_embed_local:embed/2

Handle edge cases: empty vector [[]], no embeddings [], and unexpected result formats

Added

Unit tests for embed error handling in barrel_vectordb_embed_local_tests

[1.2.4] - 2026-01-05

Changed

Update rocksdb dependency from 2.2.0 to 2.4.1

[1.2.3] - 2026-01-04

Fixed

Fix CI cache configuration: invalidate old cache and use proper path expansion

[1.2.2] - 2026-01-04

Fixed

Fix dialyzer warnings in barrel_vectordb_models (dead code removal, type spec update)

Fix dialyzer warnings in barrel_vectordb_python_queue (unmatched return values)

Exclude optional barrel_vectordb_index_faiss module from dialyzer checks

[1.2.1] - 2026-01-04

Fixed

Add missing gen_batch_server to application dependencies in app.src

[1.2.0] - 2026-01-04

Added

FAISS backend: Optional high-performance vector indexing via barrel_faiss

Pluggable backend architecture: New barrel_vectordb_index behaviour for index backends

Backend selection: Choose backend at store initialization with #{backend => hnsw | faiss}

FAISS features:

HNSW32 index type (fast approximate search)

Soft delete with compact/rebuild support

Cosine and Euclidean distance functions

Full serialization/deserialization support

Backend benchmarks: New barrel_vectordb_backend_bench module for comparing backends

Benchmark script: scripts/run_backend_bench.sh for easy performance testing

Performance

FAISS vs HNSW benchmark results (64 dimensions, 500 vectors):

Operation | HNSW | FAISS | Winner
insert_single | 14.5K ops/s | 23.8K ops/s | FAISS 1.6x
insert_batch_100 | 3.6K ops/s | 11.0K ops/s | FAISS 3.1x
search_k10 | 2.2K ops/s | 5.0K ops/s | FAISS 2.2x
index_build_1k | 729 ops/s | 4.7K ops/s | FAISS 6.5x
delete_single | 34.7K ops/s | 19.4K ops/s | HNSW 1.8x

Changed

State record: hnsw_index renamed to index in server state

Stats output: Index info now under index key instead of hnsw

[1.1.0] - 2026-01-01

Added

Search options: include_text and include_metadata options to skip unnecessary RocksDB lookups

Search option: ef_search option to control search width at query time

Batch vector API: add_vector_batch/2 for efficient bulk vector insertion

Checkpoint API: checkpoint/1 for manual HNSW index persistence

gen_batch_server integration: Automatic write batching for improved throughput

Benchmark framework: Performance benchmark suite in bench/ directory

Multi-writer tests: Concurrent writer stress tests

Improved documentation: Comprehensive README with all API functions and options

Changed

HNSW search optimization: Replaced lists:sort/1 with gb_trees for O(log N) candidate management instead of O(N log N)

Batch RocksDB lookups: Search now uses rocksdb:multi_get/4 instead of sequential rocksdb:get/3 calls

Vector storage: Use float32 for vector storage (50% size reduction)

HNSW persistence: Deferred HNSW persistence for faster inserts

Dependencies: Updated rocksdb from 2.0.0 to 2.2.0 for multi_get support

Fixed

Benchmark warmup: Fixed store reset between warmup and actual benchmark runs

Search latency variance: Reduced max search latency from 670ms to sub-10ms by fixing cold-start and optimizing lookups

Performance

These changes significantly reduced search latency variance:

Metric | Before | After
P50 | 1.3ms | ~1ms
Max | 670ms | <10ms
Variance | 500x | <10x

Key optimizations:

Fixed benchmark warmup keeping HNSW index warm

Batch RocksDB lookups with multi_get (2 calls per result -> 2 total)

Skip text/metadata lookups with include_text => false

O(log N) HNSW candidate management with gb_trees

[1.0.0] - 2025-12-01

Added

Initial release

RocksDB-backed vector storage with column families

HNSW approximate nearest neighbor search

8-bit vector quantization with norm caching

Pluggable embedding providers:

Local (Python sentence-transformers)

Ollama

OpenAI

Provider chain for fallback

Metadata filtering on search

GitLab CI configuration

HexDocs integration

Features

barrel_vectordb:add/4 - Add document with text embedding

barrel_vectordb:add_vector/5 - Add document with pre-computed vector

barrel_vectordb:search/3 - Text-based semantic search

barrel_vectordb:search_vector/3 - Vector-based search

barrel_vectordb:get/2 - Retrieve document by ID

barrel_vectordb:update/4 - Update existing document

barrel_vectordb:upsert/4 - Insert or update document

barrel_vectordb:delete/2 - Delete document

barrel_vectordb:peek/2 - Sample random documents

barrel_vectordb:count/1 - Count total documents

LICENSE

                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to the Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   Copyright 2025 Barrel-DB

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

barrel_vectordb

barrel_vectordb - Erlang Vector Database

An Erlang library for storing and searching vectors. Supports optional embedding providers for text-to-vector conversion.

Quick Start (with embeddings)

   %% Start a store with local Python embeddings
   {ok, _} = barrel_vectordb:start_link(#{
       name => my_store,
       path => "/tmp/vectors",
       embedder => {local, #{}}  %% requires Python + sentence-transformers
   }).

   %% Add a document (embeds the text automatically)
   ok = barrel_vectordb:add(my_store, <<"doc-1">>, <<"Hello world">>, #{}).

   %% Search with text query
   {ok, Results} = barrel_vectordb:search(my_store, <<"greetings">>, #{k => 5}).

Quick Start (vector-only, no embedder)

   %% Start a store without embedder
   {ok, _} = barrel_vectordb:start_link(#{
       name => my_store,
       path => "/tmp/vectors",
       dimensions => 768
   }).

   %% Add with pre-computed vector
   ok = barrel_vectordb:add_vector(my_store, <<"doc-1">>, <<"Hello">>, #{}, Vector).

   %% Search with vector query
   {ok, Results} = barrel_vectordb:search_vector(my_store, QueryVector, #{k => 5}).

Configuration

   #{
       name => atom() | binary(),   %% Store name (required)
       path => string(),            %% RocksDB path
       dimensions => pos_integer(), %% Vector dimensions (default: 768)
       embedder => EmbedderConfig,  %% Embedding provider (optional)
       hnsw => HnswConfig           %% HNSW index parameters
   }

Embedding Providers

Embedder is **explicit** - if not configured, only add_vector/5 and search_vector/3 work. Text-based operations return {error, embedder_not_configured}.

   %% Local Python with sentence-transformers (CPU, no external calls)
   embedder => {local, #{
       python => "python3",
       model => "BAAI/bge-base-en-v1.5"
   }}

   %% Ollama (local LLM server)
   embedder => {ollama, #{
       url => <<"http://localhost:11434">>,
       model => <<"nomic-embed-text">>
   }}

   %% Provider chain with fallback
   embedder => [
       {ollama, #{url => <<"http://localhost:11434">>}},
       {local, #{}}  %% Fallback to CPU
   ]

Summary

Types

diskann_config/0

DiskANN index parameters.

embedder_config/0

Embedding provider configuration.

faiss_config/0

FAISS index parameters.

hnsw_config/0

HNSW index parameters.

id/0

Unique document identifier.

metadata/0

Arbitrary metadata map associated with a document.

search_opts/0

search_result/0

A single search result.

store/0

A store reference - a registered name (binary preferred; atoms are normalized, so dynamic names should be binaries) or a pid.

store_config/0

text/0

Document text content.

vector/0

Embedding vector (list of floats).

Functions

add(Store, Id, Text, Metadata)

Add a document with automatic embedding.

add(Store, Id, Text, Metadata, Vector)

Add a document with explicit vector.

add_batch(Store, Docs)

Add multiple documents in batch.

add_index_only(Store, Id, Text, Vector)

Index a vector without storing text or metadata.

add_index_only_batch(Store, Entries)

Index multiple vectors without storing text or metadata.

add_vector(Store, Id, Text, Metadata, Vector)

Add a document with pre-computed vector (alias).

add_vector_batch(Store, Docs)

Add multiple documents with pre-computed vectors in batch.

checkpoint(Store)

Checkpoint the HNSW index to disk.

count(Store)

Get document count.

delete(Store, Id)

Delete a document.

destroy(Store)

Destroy a store: close every handle (RocksDB, index, disk BM25) and remove the storage directory. The store stops.

embed(Store, Text)

Generate embedding for a single text.

embed_batch(Store, Texts)

Generate embeddings for multiple texts.

embedder_info(Store)

Get embedding provider information.

get(Store, Id)

Get a document by ID.

peek(Store, Limit)

Peek at documents.

search(Store, Query, Opts)

Search for similar documents using text query.

search_bm25(Store, Query, Opts)

Search for similar documents using BM25 text search.

search_hybrid(Store, Query, Opts)

Hybrid search combining BM25 and vector search.

search_vector(Store, Vector, Opts)

Search for similar documents using a vector query.

start_link(Config)

Start a new vector store.

stats(Store)

Get store statistics.

stop(Store)

Stop a vector store.

update(Store, Id, Text, Metadata)

Update a document.

upsert(Store, Id, Text, Metadata)

Insert or update a document.

Types

diskann_config/0

-type diskann_config() :: #{base_path => string() | binary(), l => pos_integer(), r => pos_integer()}.

DiskANN index parameters.

embedder_config/0

-type embedder_config() ::
          {local, map()} | {ollama, map()} | {openai, map()} | {anthropic, map()} | [{atom(), map()}].

Embedding provider configuration.

faiss_config/0

-type faiss_config() :: #{index_type => binary(), nprobe => pos_integer()}.

FAISS index parameters.

hnsw_config/0

-type hnsw_config() ::
          #{m => pos_integer(), ef_construction => pos_integer(), distance_fn => cosine | euclidean}.

HNSW index parameters.

id/0

-type id() :: binary().

Unique document identifier.

metadata/0

-type metadata() :: #{atom() => term()}.

Arbitrary metadata map associated with a document.

search_opts/0

-type search_opts() ::
          #{k => pos_integer(),
            filter => fun((metadata()) -> boolean()),
            include_text => boolean(),
            include_metadata => boolean(),
            ef_search => pos_integer(),
            query_vector => [number()],
            bm25_weight => float(),
            vector_weight => float(),
            fusion => rrf | linear}.

search_result/0

-type search_result() ::
          #{key := id(), text := text(), metadata := metadata(), score := float(), vector => vector()}.

A single search result.

store/0

-type store() :: atom() | binary() | pid().

A store reference - a registered name (binary preferred; atoms are normalized, so dynamic names should be binaries) or a pid.

store_config/0

-type store_config() ::
          #{name := atom() | binary(),
            path => string() | binary(),
            db_path => string() | binary(),
            dimension => pos_integer(),
            dimensions => pos_integer(),
            embedder => embedder_config(),
            backend => hnsw | faiss | diskann,
            hnsw => hnsw_config(),
            faiss => faiss_config(),
            diskann => diskann_config(),
            bm25_backend => memory | disk | none,
            bm25 => map(),
            bm25_disk => map(),
            docstore => {module(), map()},
            _ => _}.

text/0

-type text() :: binary().

Document text content.

vector/0

-type vector() :: [float()].

Embedding vector (list of floats).

Functions

add(Store, Id, Text, Metadata)

-spec add(store(), id(), text(), metadata()) -> ok | {error, term()}.

Add a document with automatic embedding.

Embeds the text using the configured provider and stores it in the vector database along with its metadata.

Example

  ok = barrel_vectordb:add(my_store, <<"doc-1">>, <<"Hello world">>, #{
      type => greeting,
      language => english
  }).

add(Store, Id, Text, Metadata, Vector)

-spec add(store(), id(), text(), metadata(), vector()) -> ok | {error, term()}.

Add a document with explicit vector.

Stores the document with a pre-computed embedding vector instead of generating one automatically.

Example

  Vector = [0.1, 0.2, ...],  %% 768 dimensions
  ok = barrel_vectordb:add(my_store, <<"doc-1">>, <<"Hello">>, #{}, Vector).

add_batch(Store, Docs)

-spec add_batch(store(), [{id(), text(), metadata()}]) ->
                   {ok, #{inserted := non_neg_integer()}} | {error, term()}.

Add multiple documents in batch.

Efficiently adds multiple documents with automatic embedding. More efficient than calling add/4 multiple times.

Example

  Docs = [
      {<<"id-1">>, <<"text 1">>, #{type => a}},
      {<<"id-2">>, <<"text 2">>, #{type => b}}
  ],
  {ok, #{inserted := 2}} = barrel_vectordb:add_batch(my_store, Docs).

add_index_only(Store, Id, Text, Vector)

-spec add_index_only(store(), id(), text(), vector()) -> ok | {error, term()}.

Index a vector without storing text or metadata.

For callers that own document storage elsewhere (for example a document database fronted by a barrel_vectordb_docstore adapter): the vector is written to the vector column family in the atomic batch (it stays authoritative for index rebuild on restart), Text feeds the BM25 index transiently and is then dropped, and no text/metadata is stored anywhere. Any stale text/metadata rows from a previous full add are cleared in the same batch.

Idempotent upsert by id: re-adding replaces the vector and the BM25 entry, so a crashed producer can safely re-drive the same entries. Use delete/2 to remove an entry.

add_index_only_batch(Store, Entries)

-spec add_index_only_batch(store(), [{id(), text(), vector()}]) ->
                              {ok, #{inserted := non_neg_integer()}} | {error, term()}.

Index multiple vectors without storing text or metadata.

Batch form of add_index_only/4: one atomic RocksDB write for all vectors.

add_vector(Store, Id, Text, Metadata, Vector)

-spec add_vector(store(), id(), text(), metadata(), vector()) -> ok | {error, term()}.

Add a document with pre-computed vector (alias).

Same as add/5, provided for API clarity.

See also: add/5.

add_vector_batch(Store, Docs)

-spec add_vector_batch(store(), [{id(), text(), metadata(), vector()}]) ->
                          {ok, #{inserted := non_neg_integer()}} | {error, term()}.

Add multiple documents with pre-computed vectors in batch.

Efficiently adds multiple documents with their vectors in a single atomic RocksDB write. Much faster than calling add_vector/5 multiple times.

Example

  Docs = [
      {<<"id-1">>, <<"text 1">>, #{type => a}, Vector1},
      {<<"id-2">>, <<"text 2">>, #{type => b}, Vector2}
  ],
  {ok, #{inserted := 2}} = barrel_vectordb:add_vector_batch(my_store, Docs).

checkpoint(Store)

-spec checkpoint(store()) -> ok.

Checkpoint the HNSW index to disk.

Persists the current in-memory HNSW index metadata to RocksDB. This can speed up restart by avoiding a full index rebuild.

Note: The index is automatically rebuilt from vectors on startup if no checkpoint exists, so this is optional but improves restart time.

Example

  ok = barrel_vectordb:checkpoint(my_store).

count(Store)

-spec count(store()) -> non_neg_integer().

Get document count.

Returns the number of documents in the store.

delete(Store, Id)

-spec delete(store(), id()) -> ok | {error, term()}.

Delete a document.

Removes a document from the store, including its vector and metadata.

Example

  ok = barrel_vectordb:delete(my_store, <<"doc-1">>).

destroy(Store)

-spec destroy(store()) -> ok | {error, term()}.

Destroy a store: close every handle (RocksDB, index, disk BM25) and remove the storage directory. The store stops.

embed(Store, Text)

-spec embed(store(), text()) -> {ok, vector()} | {error, term()}.

Generate embedding for a single text.

Uses the store's configured embedding provider to generate a vector. Useful when you need the vector for other purposes.

Example

  {ok, Vector} = barrel_vectordb:embed(my_store, <<"hello world">>),
  768 = length(Vector).

embed_batch(Store, Texts)

-spec embed_batch(store(), [text()]) -> {ok, [vector()]} | {error, term()}.

Generate embeddings for multiple texts.

More efficient than calling embed/2 multiple times as it batches the requests to the embedding provider.

Example

  {ok, Vectors} = barrel_vectordb:embed_batch(my_store, [<<"text 1">>, <<"text 2">>]),
  2 = length(Vectors).

embedder_info(Store)

-spec embedder_info(store()) -> {ok, map()}.

Get embedding provider information.

Returns information about the configured embedding providers.

Example

  {ok, Info} = barrel_vectordb:embedder_info(my_store),
  Dimension = maps:get(dimension, Info),
  Providers = maps:get(providers, Info).

get(Store, Id)

-spec get(store(), id()) -> {ok, map()} | not_found | {error, term()}.

Get a document by ID.

Retrieves a document with its vector, text, and metadata.

Example

  {ok, Doc} = barrel_vectordb:get(my_store, <<"doc-1">>),
  Text = maps:get(text, Doc),
  Meta = maps:get(metadata, Doc).

peek(Store, Limit)

-spec peek(store(), pos_integer()) -> {ok, [map()]}.

Peek at documents.

Returns a sample of documents without performing a search. Useful for inspecting the store contents.

Example

  {ok, Docs} = barrel_vectordb:peek(my_store, 10),
  [#{key := K, text := T, metadata := M} | _] = Docs.

search(Store, Query, Opts)

-spec search(store(), text(), search_opts()) -> {ok, [search_result()]} | {error, term()}.

Search for similar documents using text query.

Embeds the query text and finds the most similar documents using approximate nearest neighbor search (HNSW).

Example

  {ok, Results} = barrel_vectordb:search(my_store, <<"hello">>, #{
      k => 5,
      filter => fun(Meta) -> maps:get(type, Meta, undefined) =:= greeting end
  }),
  [#{key := Key, text := Text, score := Score} | _] = Results.

search_bm25(Store, Query, Opts)

-spec search_bm25(store(), text(), search_opts()) -> {ok, [{id(), float()}]} | {error, term()}.

Search for similar documents using BM25 text search.

Performs keyword-based BM25 text search. Requires BM25 backend to be enabled in the store configuration.

Example

  {ok, Results} = barrel_vectordb:search_bm25(my_store, <<"erlang programming">>, #{
      k => 10
  }),
  [{DocId, Score} | _] = Results.

search_hybrid(Store, Query, Opts)

-spec search_hybrid(store(), text(), search_opts()) -> {ok, [search_result()]} | {error, term()}.

Hybrid search combining BM25 and vector search.

Performs both BM25 text search and vector similarity search, then combines the results using a fusion algorithm (RRF or linear).

Example

  {ok, Results} = barrel_vectordb:search_hybrid(my_store, <<"erlang">>, #{
      k => 10,
      bm25_weight => 0.5,
      vector_weight => 0.5,
      fusion => rrf  %% or 'linear'
  }).

Results carry text and metadata like vector search (disable with include_text => false / include_metadata => false). Passing query_vector => Vector skips the internal embedder for the vector leg, so stores without an embedder can run hybrid search when the caller embeds the query itself.

search_vector(Store, Vector, Opts)

-spec search_vector(store(), vector(), search_opts()) -> {ok, [search_result()]} | {error, term()}.

Search for similar documents using a vector query.

Finds the most similar documents to the given vector without needing to embed a text query first.

Example

  QueryVector = [0.1, 0.2, ...],  %% 768 dimensions
  {ok, Results} = barrel_vectordb:search_vector(my_store, QueryVector, #{k => 5}).

start_link(Config)

-spec start_link(store_config()) -> {ok, pid()} | {error, term()}.

Start a new vector store.

Creates a new vector store with the given configuration. The store is registered under the name provided in the config.

Example

  {ok, Pid} = barrel_vectordb:start_link(#{
      name => my_store,
      path => "/var/lib/myapp/vectors",
      dimensions => 768,
      embedder => {local, #{}}
  }).

stats(Store)

-spec stats(store()) -> {ok, map()}.

Get store statistics.

Returns information about the store including document count, dimensions, and HNSW index statistics.

Example

  {ok, Stats} = barrel_vectordb:stats(my_store),
  Count = maps:get(count, Stats),
  Dims = maps:get(dimension, Stats).

stop(Store)

-spec stop(store()) -> ok.

Stop a vector store.

Gracefully shuts down the store, persisting any pending data.

update(Store, Id, Text, Metadata)

-spec update(store(), id(), text(), metadata()) -> ok | not_found | {error, term()}.

Update a document.

Updates an existing document by re-embedding the text and storing the new text and metadata. Returns not_found if the document does not exist.

Example

  ok = barrel_vectordb:update(my_store, <<"doc-1">>, <<"New text">>, #{updated => true}).

upsert(Store, Id, Text, Metadata)

-spec upsert(store(), id(), text(), metadata()) -> ok | {error, term()}.

Insert or update a document.

If the document exists, updates it. If not, inserts it. Always succeeds (unless there's an error).

Example

  ok = barrel_vectordb:upsert(my_store, <<"doc-1">>, <<"Text">>, #{}).

barrel_vectordb_app

barrel_vectordb application module

This module implements the OTP application behaviour for barrel_vectordb. It starts the top-level supervisor. barrel_vectordb is an embedded library; vector stores are started on demand via barrel_vectordb:start_link/1.

barrel_vectordb_bm25

BM25 sparse embeddings (pure Erlang, in-memory)

Implements BM25 (Best Matching 25) algorithm for sparse text retrieval. No external dependencies - runs entirely in Erlang.

Important: In-Memory Index

This module maintains an **in-memory** index. The index is not persisted and will be lost when the process terminates. This design is intentional:

- Simple and fast for small/medium datasets - Suitable for hybrid search (combine with dense vectors) - Can be rebuilt from documents stored in barrel_vectordb on startup - No additional storage overhead

For large datasets requiring persistence, consider rebuilding the index from your document store on application startup.

Use Cases

- Hybrid search: combine BM25 keyword scores with dense vector similarity - Re-ranking: use BM25 to re-rank dense search results - Keyword filtering: pre-filter candidates before expensive dense search - Small datasets: where the index comfortably fits in memory

Usage

   %% Create a new BM25 index
   Index = barrel_vectordb_bm25:new().

   %% Add documents
   Index1 = barrel_vectordb_bm25:add(Index, <<"doc1">>, <<"Hello world">>).
   Index2 = barrel_vectordb_bm25:add(Index1, <<"doc2">>, <<"Hello there">>).

   %% Search
   Results = barrel_vectordb_bm25:search(Index2, <<"hello">>, 10).
   %% => [{<<"doc1">>, 0.85}, {<<"doc2">>, 0.82}]

   %% Get sparse vector for a document
   {ok, SparseVec} = barrel_vectordb_bm25:get_vector(Index2, <<"doc1">>).
   %% => #{<<"hello">> => 0.5, <<"world">> => 0.8}

   %% Hybrid search example (combine with dense vectors)
   DenseResults = barrel_vectordb:search(Store, Query, #{k => 100}),
   BM25Results = barrel_vectordb_bm25:search(Index, Query, 100),
   HybridResults = combine_scores(DenseResults, BM25Results, 0.5).

Algorithm

BM25 score = sum(IDF(t) * TF(t,d) * (k1 + 1) / (TF(t,d) + k1 * (1 - b + b * |d| / avgdl)))

Where: - IDF(t) = log((N - n(t) + 0.5) / (n(t) + 0.5) + 1) - TF(t,d) = term frequency of t in document d - N = total number of documents - n(t) = number of documents containing term t - |d| = length of document d (in tokens) - avgdl = average document length - k1 = 1.2 (term frequency saturation parameter) - b = 0.75 (length normalization parameter)

Summary

Types

bm25_config/0

bm25_index/0

doc_info/0

sparse_vector/0

vocab_info/0

Functions

add(Index, DocId, Text)

Add a document to the index.

encode(Index, Text)

Encode text into a sparse vector without adding to index. Uses current index statistics for IDF calculation.

get_vector(Index, DocId)

Get the sparse vector representation of a document. Returns term weights based on BM25 scoring.

new()

Create a new empty BM25 index with default parameters.

new(Config)

Create a new empty BM25 index with custom parameters.

remove(Index, DocId)

Remove a document from the index.

search(Index, Query, K)

Search the index with a query. Returns list of {DocId, Score} sorted by score descending.

stats(Index)

Get index statistics.

Types

bm25_config/0

-type bm25_config() ::
          #{k1 := float(), b := float(), min_term_length := pos_integer(), lowercase := boolean()}.

bm25_index/0

-type bm25_index() ::
          #{docs := #{binary() => doc_info()},
            vocab := #{binary() => vocab_info()},
            total_docs := non_neg_integer(),
            total_tokens := non_neg_integer(),
            config := bm25_config()}.

doc_info/0

-type doc_info() :: #{terms := #{binary() => pos_integer()}, length := non_neg_integer()}.

sparse_vector/0

-type sparse_vector() :: #{binary() => float()}.

vocab_info/0

-type vocab_info() :: #{doc_freq := non_neg_integer()}.

Functions

add(Index, DocId, Text)

-spec add(bm25_index(), binary(), binary()) -> bm25_index().

Add a document to the index.

encode(Index, Text)

-spec encode(bm25_index(), binary()) -> sparse_vector().

Encode text into a sparse vector without adding to index. Uses current index statistics for IDF calculation.

get_vector(Index, DocId)

-spec get_vector(bm25_index(), binary()) -> {ok, sparse_vector()} | {error, not_found}.

Get the sparse vector representation of a document. Returns term weights based on BM25 scoring.

new()

-spec new() -> bm25_index().

Create a new empty BM25 index with default parameters.

new(Config)

-spec new(map()) -> bm25_index().

Create a new empty BM25 index with custom parameters.

remove(Index, DocId)

-spec remove(bm25_index(), binary()) -> bm25_index().

Remove a document from the index.

search(Index, Query, K)

-spec search(bm25_index(), binary(), pos_integer()) -> [{binary(), float()}].

Search the index with a query. Returns list of {DocId, Score} sorted by score descending.

stats(Index)

-spec stats(bm25_index()) -> map().

Get index statistics.

barrel_vectordb_bm25_disk

Disk-Native BM25 Backend

Implements BM25 text search with disk-native storage using: - Block-Max MaxScore algorithm for early termination - Hot layer for fast writes, background compaction to disk - RocksDB for term/doc ID mapping - mmap for block-max index reads

Architecture: - Hot layer (RAM): Recent documents, sub-ms write latency - Disk layer (SSD): Compressed postings with block-max index - RocksDB: term string <-> int ID, doc string <-> int ID mapping - ETS: Doc lengths and stats (small, hot)

Summary

Types

bm25_disk_index/0

sparse_vector/0

Functions

add(Index, DocId, Text)

Add a document to the index

build(Index, Rest)

Build index from list of documents Docs format: [{DocId, Text}, ...]

close(Bm25_disk_index)

Close the index

compact(Bm25_disk_index)

Compact hot layer to disk

encode(Index, Text)

Encode text into sparse vector without adding to index

get_vector(Index, DocId)

Get sparse vector representation of a document

info(Index)

Get detailed index info

new(Options)

Create a new disk-native BM25 index Options: - base_path: Path for disk storage (required) - k1: BM25 k1 parameter (default: 1.2) - b: BM25 b parameter (default: 0.75) - min_term_length: Minimum token length (default: 1) - lowercase: Lowercase tokens (default: true) - hot_max_size: Max docs in hot layer (default: 50000) - hot_compaction_threshold: Trigger compaction at this % (default: 0.8) - block_size: Documents per posting block (default: 128)

open(Path)

Open an existing disk-native BM25 index

open(Path, Opts)

Open with options: crypto => none | #{key := <<_:256>>, env => rocksdb env}. The key encrypts the flat files; the env is the EncryptedEnv for the bm25.ids RocksDB.

remove(Index, DocId)

Remove a document from the index

search(Index, Query, K)

Search the index

search_with_metrics(Index, Query, K)

Search the index and return metrics for debugging/tuning Returns {Results, Metrics} where Metrics includes block skip statistics

stats(Bm25_disk_index)

Get index statistics

sync(Bm25_disk_index)

Sync to disk

Types

bm25_disk_index/0

-type bm25_disk_index() ::
          #bm25_disk_index{config ::
                               #bm25_disk_config{k1 :: float(),
                                                 b :: float(),
                                                 min_term_length :: pos_integer(),
                                                 lowercase :: boolean(),
                                                 block_size :: pos_integer()},
                           base_path :: binary(),
                           total_docs :: non_neg_integer(),
                           total_tokens :: non_neg_integer(),
                           next_term_int_id :: non_neg_integer(),
                           next_doc_int_id :: non_neg_integer(),
                           id_db :: rocksdb:db_handle() | undefined,
                           cf_terms_fwd :: rocksdb:cf_handle() | undefined,
                           cf_terms_rev :: rocksdb:cf_handle() | undefined,
                           cf_docs_fwd :: rocksdb:cf_handle() | undefined,
                           cf_docs_rev :: rocksdb:cf_handle() | undefined,
                           id_db_standalone :: boolean(),
                           id_db_env :: term() | undefined,
                           file_handle :: term() | undefined,
                           doc_stats_table :: ets:tid() | undefined,
                           term_stats_table :: ets:tid() | undefined,
                           hot_enabled :: boolean(),
                           hot_max_size :: non_neg_integer(),
                           hot_compaction_threshold :: float(),
                           hot_postings :: #{non_neg_integer() => [{non_neg_integer(), pos_integer()}]},
                           hot_docs :: #{non_neg_integer() => #{non_neg_integer() => pos_integer()}},
                           hot_doc_lengths :: #{non_neg_integer() => non_neg_integer()},
                           hot_size :: non_neg_integer(),
                           hot_tokens :: non_neg_integer(),
                           disk_doc_count :: non_neg_integer(),
                           disk_term_count :: non_neg_integer(),
                           disk_total_tokens :: non_neg_integer(),
                           blockmax_index :: #{non_neg_integer() => [map()]},
                           compaction_in_progress :: boolean()}.

sparse_vector/0

-type sparse_vector() :: #{binary() => float()}.

Functions

add(Index, DocId, Text)

-spec add(bm25_disk_index(), binary(), binary()) -> {ok, bm25_disk_index()} | {error, term()}.

Add a document to the index

build(Index, Rest)

-spec build(bm25_disk_index(), [{binary(), binary()}]) -> {ok, bm25_disk_index()} | {error, term()}.

Build index from list of documents Docs format: [{DocId, Text}, ...]

close(Bm25_disk_index)

-spec close(bm25_disk_index()) -> ok.

Close the index

compact(Bm25_disk_index)

-spec compact(bm25_disk_index()) -> {ok, bm25_disk_index()} | {error, term()}.

Compact hot layer to disk

encode(Index, Text)

-spec encode(bm25_disk_index(), binary()) -> sparse_vector().

Encode text into sparse vector without adding to index

get_vector(Index, DocId)

-spec get_vector(bm25_disk_index(), binary()) -> {ok, sparse_vector()} | {error, not_found}.

Get sparse vector representation of a document

info(Index)

-spec info(bm25_disk_index()) -> map().

Get detailed index info

new(Options)

-spec new(map()) -> {ok, bm25_disk_index()} | {error, term()}.

Create a new disk-native BM25 index Options: - base_path: Path for disk storage (required) - k1: BM25 k1 parameter (default: 1.2) - b: BM25 b parameter (default: 0.75) - min_term_length: Minimum token length (default: 1) - lowercase: Lowercase tokens (default: true) - hot_max_size: Max docs in hot layer (default: 50000) - hot_compaction_threshold: Trigger compaction at this % (default: 0.8) - block_size: Documents per posting block (default: 128)

open(Path)

-spec open(binary() | string()) -> {ok, bm25_disk_index()} | {error, term()}.

Open an existing disk-native BM25 index

open(Path, Opts)

-spec open(binary() | string(), map()) -> {ok, bm25_disk_index()} | {error, term()}.

Open with options: crypto => none | #{key := <<_:256>>, env => rocksdb env}. The key encrypts the flat files; the env is the EncryptedEnv for the bm25.ids RocksDB.

remove(Index, DocId)

-spec remove(bm25_disk_index(), binary()) -> {ok, bm25_disk_index()} | {error, not_found}.

Remove a document from the index

search(Index, Query, K)

-spec search(bm25_disk_index(), binary(), pos_integer()) -> [{binary(), float()}].

Search the index

search_with_metrics(Index, Query, K)

-spec search_with_metrics(bm25_disk_index(), binary(), pos_integer()) -> {[{binary(), float()}], map()}.

Search the index and return metrics for debugging/tuning Returns {Results, Metrics} where Metrics includes block skip statistics

stats(Bm25_disk_index)

-spec stats(bm25_disk_index()) -> map().

Get index statistics

sync(Bm25_disk_index)

-spec sync(bm25_disk_index()) -> ok.

Sync to disk

barrel_vectordb_bm25_disk_file

BM25 Disk File I/O

Handles persistent storage for disk-native BM25 index with: - Varint encoding for compact posting storage - 4KB sector-aligned writes for optimal SSD performance - Block-max index for early termination - mmap support for zero-copy reads

File layout: - bm25.meta: Header + global stats (4KB) - bm25.postings: Compressed posting blocks (4KB aligned) - bm25.blockmax: Block-max index (mmap'd) - bm25.terms/: RocksDB for term <-> int mapping - bm25.docs/: RocksDB for doc <-> int mapping

Summary

Types

block_max_entry/0

bm25_file/0

posting/0

Functions

close(Bm25_file)

Close file handles

create(Path, Config)

Create a new BM25 disk index

decode_posting_block(Bin)

Decode a posting block Returns list of {DocIntId, TF}

encode_posting_block(Postings)

Encode a block of postings using delta encoding + varint Input: [{DocIntId, TF}, ...] sorted by DocIntId ascending Output: Binary with format: [Count:varint][Delta1:varint][TF1:varint]...

get_path(Bm25_file)

Get the base path

has_mmap(Bm25_file)

Check if file has mmap handle

lookup_blocks_for_term(File, TermIntId)

Lookup blocks for a specific term

mmap_close(Mmap)

Close mmap handle

mmap_open(Path)

Open a file with mmap for zero-copy reads

mmap_read(Mmap, Offset, Size)

Read from mmap

open(Path)

Open an existing BM25 disk index

open(Path, Opts)

Open with options: crypto => none | #{key := <<_:256>>}. The open fails closed on any encrypted/plaintext mismatch or wrong key.

pad_to_sector(Bin)

Pad binary to sector alignment

read_block(Bm25_file, Offset)

Read a block from the specified offset

read_blockmax_index(Bm25_file)

Read block-max index from file Returns map of TermIntId => [BlockMaxEntry]

read_blocks(Bm25_file, OffsetSizePairs)

Read multiple blocks (batch read)

read_header(Bm25_file)

Read index header (returns cached header)

read_postings(Bm25_file, Offset, Size)

Read postings from specified offset and size

rotate_data_nonce(Bm25_file)

Fresh postings nonce for a full rewrite (compaction): posting offsets are reused across compactions, so the static-mode nonce must rotate before the rewrite. Persisted by the next header write.

sync(Bm25_file)

Sync all file handles to disk

update_stats(Bm25_file, Stats)

Update statistics in header

varint_decode(Bin)

Decode a varint from the beginning of a binary Returns {Value, Rest} where Rest is the remaining binary

varint_decode_list(Bin, Count)

Decode multiple varints from a binary Returns list of values

varint_encode(N)

Encode an unsigned integer as a varint (7-bit encoding) Each byte uses 7 bits for data and 1 bit as continuation flag

varint_encode_list(Values)

Encode a list of integers as varints

write_block(Bm25_file, Offset, Data)

Write a block at the specified offset (sector-aligned)

write_blockmax_index(Bm25_file, TermBlocks)

Write block-max index to file Format: [TermCount:32][Entry1][Entry2]... Entry: [TermIntId:32][BlockCount:16][Block1]...[BlockN] Block: [MaxImpact:32/float][DocStart:32][DocEnd:32][Offset:64][Size:32]

write_header(File, Header)

Write index header

write_postings(Bm25_file, Offset, Postings)

Write postings for a term at specified offset

Types

block_max_entry/0

-type block_max_entry() ::
          #{max_impact := float(),
            doc_start := non_neg_integer(),
            doc_end := non_neg_integer(),
            offset := non_neg_integer(),
            size := non_neg_integer()}.

bm25_file/0

-type bm25_file() ::
          #bm25_file{path :: binary(),
                     meta_fd :: file:fd() | undefined,
                     postings_fd :: file:fd() | undefined,
                     blockmax_fd :: file:fd() | undefined,
                     blockmax_mmap :: term() | undefined,
                     header :: map(),
                     crypto ::
                         undefined |
                         #{key := binary(), postings_nonce := binary(), key_check := binary()}}.

posting/0

-type posting() :: {DocIntId :: non_neg_integer(), TF :: pos_integer()}.

Functions

close(Bm25_file)

-spec close(bm25_file()) -> ok.

Close file handles

create(Path, Config)

-spec create(binary() | string(), map()) -> {ok, bm25_file()} | {error, term()}.

Create a new BM25 disk index

decode_posting_block(Bin)

-spec decode_posting_block(binary()) -> {ok, [posting()]} | {error, term()}.

Decode a posting block Returns list of {DocIntId, TF}

encode_posting_block(Postings)

-spec encode_posting_block([posting()]) -> binary().

Encode a block of postings using delta encoding + varint Input: [{DocIntId, TF}, ...] sorted by DocIntId ascending Output: Binary with format: [Count:varint][Delta1:varint][TF1:varint]...

get_path(Bm25_file)

-spec get_path(bm25_file()) -> binary().

Get the base path

has_mmap(Bm25_file)

-spec has_mmap(bm25_file()) -> boolean().

Check if file has mmap handle

lookup_blocks_for_term(File, TermIntId)

-spec lookup_blocks_for_term(bm25_file(), non_neg_integer()) ->
                                {ok, [block_max_entry()]} | {error, not_found}.

Lookup blocks for a specific term

mmap_close(Mmap)

-spec mmap_close(term()) -> ok.

Close mmap handle

mmap_open(Path)

-spec mmap_open(binary() | string()) -> {ok, term()} | {error, term()}.

Open a file with mmap for zero-copy reads

mmap_read(Mmap, Offset, Size)

-spec mmap_read(term(), non_neg_integer(), non_neg_integer()) -> {ok, binary()} | {error, term()}.

Read from mmap

open(Path)

-spec open(binary() | string()) -> {ok, bm25_file()} | {error, term()}.

Open an existing BM25 disk index

open(Path, Opts)

-spec open(binary() | string(), map()) -> {ok, bm25_file()} | {error, term()}.

Open with options: crypto => none | #{key := <<_:256>>}. The open fails closed on any encrypted/plaintext mismatch or wrong key.

pad_to_sector(Bin)

-spec pad_to_sector(binary()) -> binary().

Pad binary to sector alignment

read_block(Bm25_file, Offset)

-spec read_block(bm25_file(), non_neg_integer()) -> {ok, binary()} | {error, term()}.

Read a block from the specified offset

read_blockmax_index(Bm25_file)

-spec read_blockmax_index(bm25_file()) ->
                             {ok, #{non_neg_integer() => [block_max_entry()]}} | {error, term()}.

Read block-max index from file Returns map of TermIntId => [BlockMaxEntry]

read_blocks(Bm25_file, OffsetSizePairs)

-spec read_blocks(bm25_file(), [{non_neg_integer(), non_neg_integer()}]) ->
                     {ok, [binary()]} | {error, term()}.

Read multiple blocks (batch read)

read_header(Bm25_file)

-spec read_header(bm25_file()) -> map().

Read index header (returns cached header)

read_postings(Bm25_file, Offset, Size)

-spec read_postings(bm25_file(), non_neg_integer(), non_neg_integer()) ->
                       {ok, [posting()]} | {error, term()}.

Read postings from specified offset and size

rotate_data_nonce(Bm25_file)

-spec rotate_data_nonce(bm25_file()) -> bm25_file().

Fresh postings nonce for a full rewrite (compaction): posting offsets are reused across compactions, so the static-mode nonce must rotate before the rewrite. Persisted by the next header write.

sync(Bm25_file)

-spec sync(bm25_file()) -> ok.

Sync all file handles to disk

update_stats(Bm25_file, Stats)

-spec update_stats(bm25_file(), map()) -> {ok, bm25_file()}.

Update statistics in header

varint_decode(Bin)

-spec varint_decode(binary()) -> {non_neg_integer(), binary()} | {error, incomplete}.

Decode a varint from the beginning of a binary Returns {Value, Rest} where Rest is the remaining binary

varint_decode_list(Bin, Count)

-spec varint_decode_list(binary(), non_neg_integer()) -> {[non_neg_integer()], binary()}.

Decode multiple varints from a binary Returns list of values

varint_encode(N)

-spec varint_encode(non_neg_integer()) -> binary().

Encode an unsigned integer as a varint (7-bit encoding) Each byte uses 7 bits for data and 1 bit as continuation flag

varint_encode_list(Values)

-spec varint_encode_list([non_neg_integer()]) -> binary().

Encode a list of integers as varints

write_block(Bm25_file, Offset, Data)

-spec write_block(bm25_file(), non_neg_integer(), binary()) -> ok | {error, term()}.

Write a block at the specified offset (sector-aligned)

write_blockmax_index(Bm25_file, TermBlocks)

-spec write_blockmax_index(bm25_file(), #{non_neg_integer() => [block_max_entry()]}) ->
                              {ok, bm25_file()} | {error, term()}.

Write block-max index to file Format: [TermCount:32][Entry1][Entry2]... Entry: [TermIntId:32][BlockCount:16][Block1]...[BlockN] Block: [MaxImpact:32/float][DocStart:32][DocEnd:32][Offset:64][Size:32]

write_header(File, Header)

-spec write_header(bm25_file(), map()) -> {ok, bm25_file()}.

Write index header

write_postings(Bm25_file, Offset, Postings)

-spec write_postings(bm25_file(), non_neg_integer(), [posting()]) ->
                        {ok, non_neg_integer()} | {error, term()}.

Write postings for a term at specified offset

barrel_vectordb_crypto

Encryption plumbing for vectordb stores: builds the RocksDB EncryptedEnv from a resolved key and enforces a fail-closed open matrix through a cleartext CRYPTO key-check marker inside the store directory.

The store never resolves keys itself: the barrel module resolves one key per logical database (the docdb keyspace) and passes it as crypto => #{key := <<_:256>>}; standalone users pass a key the same way. The returned env handle must stay referenced for the whole store lifetime (the NIF frees the env when the handle is garbage collected).

Summary

Types

ctx/0

Functions

init(Other, DbPath)

Types

ctx/0

-type ctx() :: none | #{key := binary(), env := rocksdb:env_handle()}.

Functions

init(Other, DbPath)

-spec init(none | #{key := binary(), _ => _}, string() | binary()) -> {ok, ctx()} | {error, term()}.

barrel_vectordb_diskann

DiskANN Vamana Graph Implementation

Implements the Vamana graph algorithm from the DiskANN paper with: - Two-pass construction (alpha=1.0 then alpha>1.0) - RobustPrune for alpha-RNG pruning - GreedySearch for graph traversal - FreshVamana insert/delete for streaming updates - Consolidate deletes for batch cleanup - BeamSearch with PQ for SSD-resident search

The alpha parameter (>1) is critical for maintaining graph quality under streaming updates. It keeps more long-range edges which are essential for fast convergence during search.

Summary

Types

diskann_config/0

diskann_index/0

diskann_node/0

Functions

build(Options, Vectors)

Build index from a list of {Id, Vector} pairs using two-pass Vamana

close(Diskann_index)

Close the index and flush to disk For V2 format: writes header and PQ state, closes RocksDB Does NOT use term_to_binary for the entire index

close_all_standalone_dbs()

compact(Diskann_index)

Trigger manual compaction of hot layer to disk

consolidate_deletes(Diskann_index)

Consolidate deleted nodes (batch cleanup) This repairs the graph by removing edges to deleted nodes and adding new edges to maintain navigability

delete(Diskann_index, Id)

Lazy delete - marks node as deleted

deserialize(Binary)

Deserialize index from binary

find_medoid(Vectors, Diskann_config)

get_vector(Diskann_index, Id)

Get vector by ID (uses cache for disk mode)

greedy_search(Index, StartId, Query, K, L)

Core search algorithm - optimized version Uses separate candidate queue and result set for efficiency Returns ({SortedResults, VisitedSet})

info(Diskann_index)

Get index info

insert(Diskann_index, Id, Vector)

Insert a new vector (FreshVamana algorithm)

insert_batch(Index, Vectors, Opts)

Batch insert multiple vectors efficiently Inserts all vectors first, then builds graph edges in one pass. Much more efficient than calling insert/3 repeatedly.

needs_consolidation(Diskann_index)

Check if consolidation is needed (threshold-based) Returns true if deleted count exceeds 10% of active size

new(Options)

Create a new empty DiskANN index Options: - dimension: Vector dimension (required) - r: Max out-degree (default: 64) - l_build: Build search width (default: 100) - l_search: Query search width (default: 100) - alpha: Pruning factor (default: 1.2) - distance_fn: cosine | euclidean (default: cosine) - assume_normalized: Skip norm computation for cosine (default: false) - storage_mode: memory | disk (default: memory) - base_path: Path for disk storage (required if storage_mode=disk) - cache_max_size: Max vectors in LRU cache (default: 10000) - hot_layer: Enable hot memory layer for fast writes (default: false) - hot_max_size: Maximum vectors in hot layer before compaction (default: 10000) - hot_compaction_threshold: Trigger compaction at this % of max (default: 0.8) - quantization: none | pq | turboquant | subspace_turboquant (default: none) - tq_bits: TurboQuant bits per component (default: 3, when quantization=turboquant or subspace_turboquant) - tq_seed: TurboQuant random seed (default: 42) - tq_m: Number of subspaces for subspace_turboquant (default: auto, based on dimension)

open(BasePath)

Open an existing DiskANN index from disk For V2 format: O(1) startup with lazy graph loading For V1 format: Migrates to V2 or loads from diskann.index

open(BasePath, Opts)

Open with options: crypto => none | #{key := <<_:256>>, env => rocksdb env}. The key encrypts the flat files; the env (or one minted from the key) covers the diskann_ids RocksDB.

robust_prune(Index, P, V, Alpha, R)

RobustPrune: Select R neighbors for node P using alpha-RNG pruning V = candidate neighbors Alpha > 1 keeps more long-range edges Optimized: cache distance computations

search(Index, Query, K)

Search for K nearest neighbors

search(Diskann_index, Query, K, Opts)

Search with options Options: - l_search: Search beam width (default: from config) - rerank_factor: Multiplier for rerank candidates (default: 4)

serialize(Diskann_index)

Serialize index to binary (for barrel_vectordb_index behaviour)

size(Diskann_index)

Get index size (excluding deleted)

sync(Diskann_index)

Force sync to disk

Types

diskann_config/0

-type diskann_config() ::
          #diskann_config{r :: pos_integer(),
                          l_build :: pos_integer(),
                          l_search :: pos_integer(),
                          alpha :: float(),
                          dimension :: pos_integer(),
                          distance_fn :: cosine | euclidean,
                          assume_normalized :: boolean()}.

diskann_index/0

-type diskann_index() ::
          #diskann_index{config ::
                             #diskann_config{r :: pos_integer(),
                                             l_build :: pos_integer(),
                                             l_search :: pos_integer(),
                                             alpha :: float(),
                                             dimension :: pos_integer(),
                                             distance_fn :: cosine | euclidean,
                                             assume_normalized :: boolean()},
                         size :: non_neg_integer(),
                         medoid_id :: binary() | undefined,
                         medoid_int_id :: non_neg_integer() | undefined,
                         nodes :: #{binary() => diskann_node()},
                         nodes_int :: #{non_neg_integer() => [non_neg_integer()]},
                         id_to_idx :: #{binary() => non_neg_integer()},
                         idx_to_id :: #{non_neg_integer() => binary()},
                         id_db :: rocksdb:db_handle() | undefined,
                         id_cf_fwd :: rocksdb:cf_handle() | undefined,
                         id_cf_rev :: rocksdb:cf_handle() | undefined,
                         id_db_standalone :: boolean(),
                         id_db_env :: term() | undefined,
                         next_int_id :: non_neg_integer(),
                         quantization_method :: pq | turboquant | subspace_turboquant | none,
                         pq_state :: term() | undefined,
                         pq_codes :: #{binary() => binary()},
                         pq_codes_int :: #{non_neg_integer() => binary()},
                         use_pq :: boolean(),
                         tq_state :: term() | undefined,
                         tq_codes :: #{binary() => binary()},
                         tq_codes_int :: #{non_neg_integer() => binary()},
                         deleted_set :: sets:set(binary()),
                         deleted_int_set :: sets:set(non_neg_integer()),
                         storage_mode :: memory | disk,
                         vectors :: #{binary() => [float()]},
                         file_handle :: term() | undefined,
                         base_path :: binary() | undefined,
                         cache_table :: ets:tid() | undefined,
                         cache_max_size :: pos_integer(),
                         graph_cache_table :: ets:tid() | undefined,
                         graph_cache_max_size :: pos_integer(),
                         hot_enabled :: boolean(),
                         hot_max_size :: non_neg_integer(),
                         hot_compaction_threshold :: float(),
                         hot_vectors :: #{non_neg_integer() => [float()]},
                         hot_graph :: #{non_neg_integer() => [non_neg_integer()]},
                         hot_id_to_int :: #{binary() => non_neg_integer()},
                         hot_int_to_id :: #{non_neg_integer() => binary()},
                         hot_next_int_id :: non_neg_integer(),
                         hot_deleted :: sets:set(non_neg_integer()),
                         hot_pq_codes :: #{non_neg_integer() => binary()},
                         hot_tq_codes :: #{non_neg_integer() => binary()},
                         hot_size :: non_neg_integer(),
                         compaction_in_progress :: boolean()}.

diskann_node/0

-type diskann_node() :: #diskann_node{id :: binary(), neighbors :: [binary()]}.

Functions

build(Options, Vectors)

-spec build(map(), [{binary(), [float()]}]) -> {ok, diskann_index()} | {error, term()}.

Build index from a list of {Id, Vector} pairs using two-pass Vamana

close(Diskann_index)

-spec close(diskann_index()) -> ok.

Close the index and flush to disk For V2 format: writes header and PQ state, closes RocksDB Does NOT use term_to_binary for the entire index

close_all_standalone_dbs()

-spec close_all_standalone_dbs() -> ok.

compact(Diskann_index)

-spec compact(diskann_index()) -> {ok, diskann_index()} | {error, term()}.

Trigger manual compaction of hot layer to disk

consolidate_deletes(Diskann_index)

-spec consolidate_deletes(diskann_index()) -> {ok, diskann_index()}.

Consolidate deleted nodes (batch cleanup) This repairs the graph by removing edges to deleted nodes and adding new edges to maintain navigability

delete(Diskann_index, Id)

-spec delete(diskann_index(), binary()) -> {ok, diskann_index()}.

Lazy delete - marks node as deleted

deserialize(Binary)

-spec deserialize(binary()) -> {ok, diskann_index()} | {error, term()}.

Deserialize index from binary

find_medoid(Vectors, Diskann_config)

get_vector(Diskann_index, Id)

-spec get_vector(diskann_index(), binary()) -> {ok, [float()]} | not_found.

Get vector by ID (uses cache for disk mode)

greedy_search(Index, StartId, Query, K, L)

Core search algorithm - optimized version Uses separate candidate queue and result set for efficiency Returns ({SortedResults, VisitedSet})

info(Diskann_index)

-spec info(diskann_index()) -> map().

Get index info

insert(Diskann_index, Id, Vector)

-spec insert(diskann_index(), binary(), [float()]) -> {ok, diskann_index()} | {error, term()}.

Insert a new vector (FreshVamana algorithm)

insert_batch(Index, Vectors, Opts)

-spec insert_batch(diskann_index(), [{binary(), [float()]}], map()) ->
                      {ok, diskann_index()} | {error, term()}.

Batch insert multiple vectors efficiently Inserts all vectors first, then builds graph edges in one pass. Much more efficient than calling insert/3 repeatedly.

needs_consolidation(Diskann_index)

-spec needs_consolidation(diskann_index()) -> boolean().

Check if consolidation is needed (threshold-based) Returns true if deleted count exceeds 10% of active size

new(Options)

-spec new(map()) -> {ok, diskann_index()} | {error, term()}.

Create a new empty DiskANN index Options: - dimension: Vector dimension (required) - r: Max out-degree (default: 64) - l_build: Build search width (default: 100) - l_search: Query search width (default: 100) - alpha: Pruning factor (default: 1.2) - distance_fn: cosine | euclidean (default: cosine) - assume_normalized: Skip norm computation for cosine (default: false) - storage_mode: memory | disk (default: memory) - base_path: Path for disk storage (required if storage_mode=disk) - cache_max_size: Max vectors in LRU cache (default: 10000) - hot_layer: Enable hot memory layer for fast writes (default: false) - hot_max_size: Maximum vectors in hot layer before compaction (default: 10000) - hot_compaction_threshold: Trigger compaction at this % of max (default: 0.8) - quantization: none | pq | turboquant | subspace_turboquant (default: none) - tq_bits: TurboQuant bits per component (default: 3, when quantization=turboquant or subspace_turboquant) - tq_seed: TurboQuant random seed (default: 42) - tq_m: Number of subspaces for subspace_turboquant (default: auto, based on dimension)

open(BasePath)

-spec open(binary() | string()) -> {ok, diskann_index()} | {error, term()}.

Open an existing DiskANN index from disk For V2 format: O(1) startup with lazy graph loading For V1 format: Migrates to V2 or loads from diskann.index

open(BasePath, Opts)

-spec open(binary() | string(), map()) -> {ok, diskann_index()} | {error, term()}.

Open with options: crypto => none | #{key := <<_:256>>, env => rocksdb env}. The key encrypts the flat files; the env (or one minted from the key) covers the diskann_ids RocksDB.

robust_prune(Index, P, V, Alpha, R)

RobustPrune: Select R neighbors for node P using alpha-RNG pruning V = candidate neighbors Alpha > 1 keeps more long-range edges Optimized: cache distance computations

search(Index, Query, K)

-spec search(diskann_index(), [float()], pos_integer()) -> [{binary(), float()}].

Search for K nearest neighbors

search(Diskann_index, Query, K, Opts)

-spec search(diskann_index(), [float()], pos_integer(), map()) -> [{binary(), float()}].

Search with options Options: - l_search: Search beam width (default: from config) - rerank_factor: Multiplier for rerank candidates (default: 4)

serialize(Diskann_index)

-spec serialize(diskann_index()) -> binary().

Serialize index to binary (for barrel_vectordb_index behaviour)

size(Diskann_index)

-spec size(diskann_index()) -> non_neg_integer().

Get index size (excluding deleted)

sync(Diskann_index)

-spec sync(diskann_index()) -> ok.

Force sync to disk

barrel_vectordb_diskann_file

DiskANN sector-aligned disk I/O

Handles persistent storage for DiskANN index with: - 4KB sector-aligned writes for optimal SSD performance - Separate files for graph, vectors, and metadata - Batch read operations for beam search

File layout: - diskann.meta: Erlang term metadata - diskann.graph: Vamana graph (4KB aligned nodes) - diskann.vectors: Full float32 vectors (4KB aligned) - diskann.pq: PQ codebooks + codes - diskann.idmap: ID to offset mapping

Summary

Types

diskann_file/0

Functions

close(Diskann_file)

Close file handles

create(Path, Config)

Create a new DiskANN index file set

get_crypto(Diskann_file)

The file set's crypto state, reduced to what companions (the pq_state term file) need: none or the key.

get_path(Diskann_file)

Get the base path

has_mmap(Diskann_file)

Check if mmap is available for zero-copy reads

open(Path)

Open an existing DiskANN index

open(Path, Opts)

Open with options: crypto => none | #{key := <<_:256>>}. The open fails closed on any encrypted/plaintext mismatch or wrong key.

read_header(Diskann_file)

Read index header (returns cached header)

read_header_from_file(Diskann_file)

Read header directly from graph file Supports both V1 (string entry point) and V2 (integer entry point) formats

read_node(Diskann_file, Offset)

Read a node from the graph file by offset

read_node_by_int_id(Diskann_file, IntId, MaxR)

Read a node from the graph file by integer ID Returns {ok, {IntId, [NeighborIntIds]}} or {error, term()}

read_nodes_batch(Diskann_file, Offsets)

Batch read multiple nodes (for beam search)

read_nodes_batch_int(Diskann_file, IntIds, MaxR)

Batch read multiple nodes by integer IDs Uses batch pread/2 for efficiency (single system call)

read_pq_codes(Diskann_file, _)

Read PQ codes for a range of vectors

read_vector(Diskann_file, Index)

Read a vector from the vector file

read_vector_mmap(Diskann_file, Index)

Read a vector using mmap (zero-copy) Falls back to regular pread if mmap is not available

sync(Diskann_file)

Sync all file handles to disk

write_header(Diskann_file, Header)

Write index header

write_node(Diskann_file, Id, Node)

Write a node to the graph file (sector-aligned) Returns {ok, Offset} where Offset is the byte offset

write_node_int(Diskann_file, IntId, Neighbors, MaxR)

Write a node to the graph file using integer ID Node format: [IntId:64/little][NeighborCount:16/little][Neighbor1:64/little]...[NeighborR:64/little][padding] Each node is stored at offset: SECTOR_SIZE + (IntId * SECTOR_SIZE)

write_pq_codes(Diskann_file, StartIndex, Codes)

Write PQ codes for a batch of vectors. Slots are written once per index (immutable append): STATIC mode.

write_vector(Diskann_file, Index, Vector)

Write a vector to the vector file. A slot is written once per index (immutable append), which is what makes STATIC mode safe here.

Types

diskann_file/0

-type diskann_file() ::
          #diskann_file{path :: binary(),
                        graph_fd :: file:fd() | undefined,
                        vector_fd :: file:fd() | undefined,
                        vector_mmap :: term() | undefined,
                        pq_fd :: file:fd() | undefined,
                        header :: map(),
                        crypto ::
                            undefined |
                            #{key := binary(),
                              vectors_nonce := binary(),
                              pq_nonce := binary(),
                              key_check := binary()}}.

Functions

close(Diskann_file)

-spec close(diskann_file()) -> ok.

Close file handles

create(Path, Config)

-spec create(binary() | string(), map()) -> {ok, diskann_file()} | {error, term()}.

Create a new DiskANN index file set

get_crypto(Diskann_file)

-spec get_crypto(diskann_file()) -> none | #{key := binary()}.

The file set's crypto state, reduced to what companions (the pq_state term file) need: none or the key.

get_path(Diskann_file)

-spec get_path(diskann_file()) -> binary().

Get the base path

has_mmap(Diskann_file)

-spec has_mmap(diskann_file()) -> boolean().

Check if mmap is available for zero-copy reads

open(Path)

-spec open(binary() | string()) -> {ok, diskann_file()} | {error, term()}.

Open an existing DiskANN index

open(Path, Opts)

-spec open(binary() | string(), map()) -> {ok, diskann_file()} | {error, term()}.

Open with options: crypto => none | #{key := <<_:256>>}. The open fails closed on any encrypted/plaintext mismatch or wrong key.

read_header(Diskann_file)

-spec read_header(diskann_file()) -> map().

Read index header (returns cached header)

read_header_from_file(Diskann_file)

-spec read_header_from_file(diskann_file()) -> {ok, map()} | {error, term()}.

Read header directly from graph file Supports both V1 (string entry point) and V2 (integer entry point) formats

read_node(Diskann_file, Offset)

-spec read_node(diskann_file(), non_neg_integer()) -> {ok, {binary(), map()}} | {error, term()}.

Read a node from the graph file by offset

read_node_by_int_id(Diskann_file, IntId, MaxR)

-spec read_node_by_int_id(diskann_file(), non_neg_integer(), pos_integer()) ->
                             {ok, {non_neg_integer(), [non_neg_integer()]}} | {error, term()}.

Read a node from the graph file by integer ID Returns {ok, {IntId, [NeighborIntIds]}} or {error, term()}

read_nodes_batch(Diskann_file, Offsets)

-spec read_nodes_batch(diskann_file(), [non_neg_integer()]) ->
                          [{ok, {binary(), map()}} | {error, term()}].

Batch read multiple nodes (for beam search)

read_nodes_batch_int(Diskann_file, IntIds, MaxR)

-spec read_nodes_batch_int(diskann_file(), [non_neg_integer()], pos_integer()) ->
                              [{ok, {non_neg_integer(), [non_neg_integer()]}} | {error, term()}].

Batch read multiple nodes by integer IDs Uses batch pread/2 for efficiency (single system call)

read_pq_codes(Diskann_file, _)

-spec read_pq_codes(diskann_file(), {non_neg_integer(), non_neg_integer()}) ->
                       {ok, [binary()]} | {error, term()}.

Read PQ codes for a range of vectors

read_vector(Diskann_file, Index)

-spec read_vector(diskann_file(), non_neg_integer()) -> {ok, [float()]} | {error, term()}.

Read a vector from the vector file

read_vector_mmap(Diskann_file, Index)

-spec read_vector_mmap(diskann_file(), non_neg_integer()) -> {ok, [float()]} | {error, term()}.

Read a vector using mmap (zero-copy) Falls back to regular pread if mmap is not available

sync(Diskann_file)

-spec sync(diskann_file()) -> ok.

Sync all file handles to disk

write_header(Diskann_file, Header)

-spec write_header(diskann_file(), map()) -> {ok, diskann_file()}.

Write index header

write_node(Diskann_file, Id, Node)

-spec write_node(diskann_file(), binary(), map()) ->
                    {ok, non_neg_integer(), diskann_file()} | {error, term()}.

Write a node to the graph file (sector-aligned) Returns {ok, Offset} where Offset is the byte offset

write_node_int(Diskann_file, IntId, Neighbors, MaxR)

-spec write_node_int(diskann_file(), non_neg_integer(), [non_neg_integer()], pos_integer()) ->
                        ok | {error, term()}.

Write a node to the graph file using integer ID Node format: [IntId:64/little][NeighborCount:16/little][Neighbor1:64/little]...[NeighborR:64/little][padding] Each node is stored at offset: SECTOR_SIZE + (IntId * SECTOR_SIZE)

write_pq_codes(Diskann_file, StartIndex, Codes)

-spec write_pq_codes(diskann_file(), non_neg_integer(), [binary()]) -> ok | {error, term()}.

Write PQ codes for a batch of vectors. Slots are written once per index (immutable append): STATIC mode.

write_vector(Diskann_file, Index, Vector)

-spec write_vector(diskann_file(), non_neg_integer(), [float()]) -> ok | {error, term()}.

Write a vector to the vector file. A slot is written once per index (immutable append), which is what makes STATIC mode safe here.

barrel_vectordb_distance

Vector Distance Computation Module

Provides SIMD-accelerated distance computations via NIF. Falls back to pure Erlang implementation if NIF is not available.

Supports: - Cosine distance - Euclidean distance - Dot product

Vectors can be represented as: - Lists of floats (Erlang native, slower) - Binaries of 32-bit floats (NIF optimized, faster)

Summary

Types

vector/0

Functions

cosine_distance(Vec1, Vec2)

Compute cosine distance: 1 - cos(a, b)

cosine_distance_normalized(Vec1, Vec2)

Compute cosine distance for pre-normalized vectors (faster)

dot_product(Vec1, Vec2)

Compute dot product of two vectors

euclidean_distance(Vec1, Vec2)

Compute Euclidean distance

from_binary(Bin)

Convert binary vector to list of floats

nif_available()

Check if NIF is available

simd_info()

Return SIMD backend information

to_binary(List)

Convert list of floats to binary vector (32-bit floats)

Types

vector/0

-type vector() :: [float()] | binary().

Functions

cosine_distance(Vec1, Vec2)

-spec cosine_distance(vector(), vector()) -> float().

Compute cosine distance: 1 - cos(a, b)

cosine_distance_normalized(Vec1, Vec2)

-spec cosine_distance_normalized(vector(), vector()) -> float().

Compute cosine distance for pre-normalized vectors (faster)

dot_product(Vec1, Vec2)

-spec dot_product(vector(), vector()) -> float().

Compute dot product of two vectors

euclidean_distance(Vec1, Vec2)

-spec euclidean_distance(vector(), vector()) -> float().

Compute Euclidean distance

from_binary(Bin)

-spec from_binary(binary()) -> [float()].

Convert binary vector to list of floats

nif_available()

-spec nif_available() -> boolean().

Check if NIF is available

simd_info()

-spec simd_info() -> #{backend => avx2 | neon | scalar | erlang}.

Return SIMD backend information

to_binary(List)

-spec to_binary([float()]) -> binary().

Convert list of floats to binary vector (32-bit floats)

barrel_vectordb_docdb_backend

barrel_docdb-backed document backend for barrel_vectordb.

Stores each vector's text and metadata as a document in a barrel_docdb database, so documents and vectors can share one source of truth (and, later, replicate together). Vectors stay in the vector store's own RocksDB; only text and metadata are kept here.

Metadata is stored as an opaque term_to_binary value so arbitrary Erlang terms (including atom keys) round-trip exactly through the document codec.

Config: #{db => binary()} (the docdb database name, default = the store name) and #{docdb_opts => map()} (passed to barrel_docdb:create_db/2).

barrel_vectordb_docstore behaviour

Document backend behaviour for barrel_vectordb.

A vector store keeps each vector's source text and metadata somewhere. By default (docstore => undefined) they live in the store's own RocksDB column families, written in the same atomic batch as the vector. Configuring an external docstore (for example barrel_vectordb_docdb_backend) routes text and metadata elsewhere; the vector itself always stays local so the ANN index can be rebuilt from it.

External backends are written after the vector batch commits (vector-first, doc-second), so a crash between the two can leave a vector without its text; the vector column family stays authoritative for rebuild.

Summary

Callbacks

delete/2

get/2

init/2

multi_get/2

multi_put/2

put/4

terminate/1

Callbacks

delete/2

-callback delete(Ctx :: term(), Id :: binary()) -> ok | {error, term()}.

get/2

-callback get(Ctx :: term(), Id :: binary()) ->
                 {ok, Text :: binary(), Metadata :: map()} | not_found | {error, term()}.

init/2

-callback init(Name :: atom() | binary(), Config :: map()) -> {ok, Ctx :: term()} | {error, term()}.

multi_get/2

-callback multi_get(Ctx :: term(), [binary()]) -> [{ok, binary(), map()} | not_found | {error, term()}].

multi_put/2

-callback multi_put(Ctx :: term(), [{binary(), binary(), map()}]) -> ok | {error, term()}.

put/4

-callback put(Ctx :: term(), Id :: binary(), Text :: binary(), Metadata :: map()) -> ok | {error, term()}.

terminate/1

-callback terminate(Ctx :: term()) -> ok.

barrel_vectordb_hnsw

HNSW (Hierarchical Navigable Small World) Implementation.

Based on the paper by Yu. A. Malkov, D. A. Yashunin: "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs" (2016).

Many implementation details not covered in the original paper were derived from antirez's practical optimizations for Redis vector-sets. Reference: https://antirez.com/news/156

Algorithm Overview

HNSW builds a multi-layer graph where each layer is a navigable small-world graph. Higher layers have fewer nodes and act as "express lanes" for fast navigation, while layer 0 contains all nodes.

Search proceeds top-down: starting from the entry point at the highest layer, we greedily descend until reaching layer 0, then perform a more thorough beam search to find the k nearest neighbors.

Insert assigns each new node to a random layer L (exponentially distributed), then connects it to neighbors at layers 0..L using the same search mechanism.

Implementation Features

1. Bidirectional links only. When a new node is inserted, we add connections from the new node to its neighbors AND from each neighbor back to the new node. This maintains graph navigability and enables proper deletion.

2. 8-bit scalar quantization. Vectors are quantized to signed int8 values using per-vector scaling:

quantized[i] = round(vector[i] * 127 / max_abs_value)

The scale factor (max_abs_value) is stored with each quantized vector. This provides 8x memory reduction compared to float64 with minimal recall loss (~1-2% for typical embeddings).

Storage format: <<Scale:32/float-little, Components/binary>> where each component is a signed 8-bit integer.

3. Norm caching. The L2 norm of each vector is computed once at insertion time and stored in the node record. This eliminates redundant norm calculations during cosine distance computation:

cosine_distance = 1 - (A·B) / (||A|| * ||B||)

Without caching: 3 dot products per comparison (A·B, A·A, B·B) With caching: 1 dot product + 2 lookups (A·B, cached norms)

4. Integer domain dot product. Distance calculations between quantized vectors use integer arithmetic:

int_dot = sum(a[i] * b[i]) -- integer multiplication real_dot = int_dot * scale_a * scale_b / (127 * 127)

This is faster than floating-point operations and SIMD-friendly.

5. Priority queue using gb_trees. The search algorithm maintains candidates in a balanced tree (Erlang's gb_trees) providing O(log N) insert and extract-min operations, versus O(N log N) for repeated lists:sort calls.

6. Direct graph serialization. The full graph structure (nodes, vectors, neighbors, metadata) is serialized to binary format for persistence. On startup, the index is deserialized directly without rebuilding the graph, providing ~100x faster loading compared to re-inserting all vectors.

Serialization format (version 2): <<Version:8, Size:32, MaxLayer:8, Dim:16, EntryPoint/binary, Config/binary, Nodes/binary>>

7. True deletion. Nodes are actually removed from the graph, not just marked as deleted. The deleted node is removed from all its neighbors' adjacency lists. A new entry point is selected if the deleted node was the entry point.

Configuration Parameters

- M: Maximum number of connections per node per layer (default: 16). Higher values improve recall but increase memory and build time.

- M_max0: Maximum connections at layer 0 (default: 2*M). Layer 0 typically needs more connections for good recall.

- ef_construction: Beam width during index construction (default: 200). Higher values improve index quality but slow down insertion.

- ef_search: Beam width during search (default: max(K, 50)). Higher values improve recall at the cost of search latency.

- ml: Level multiplier = 1/ln(M). Controls the probability distribution for assigning nodes to layers.

Complexity

- Insert: O(log N) average, O(ef_construction * M * log N) work - Search: O(log N) average for finding entry, O(ef_search * M) at layer 0 - Delete: O(M * layers) to update neighbor lists - Memory: O(N * M * layers) for graph + O(N * dim) for vectors

Summary

Types

hnsw_config/0

hnsw_index/0

hnsw_node/0

quantization_method/0

quantized_vector/0

Functions

compute_norm(Vector)

Compute L2 norm of a float vector

cosine_distance(Vec1, Vec2)

Cosine distance between two float vectors

cosine_similarity(Vec1, Vec2)

Cosine similarity between two float vectors

delete(Hnsw_index, Id)

Delete a node from the index

dequantize(_)

Dequantize back to float vector (scalar quantization)

dequantize(Quantized, Hnsw_config)

Dequantize using the method specified in config

deserialize(Binary)

Deserialize index from binary

deserialize_node(_)

Deserialize a single node

euclidean_distance(Vec1, Vec2)

Euclidean distance between two float vectors

get_node(Hnsw_index, Id)

Get a node by ID

info(Hnsw_index)

Get index info

insert(Hnsw_index, Id, Vector)

Insert a vector with given ID into the index Vector is quantized to 8-bit and norm is cached

new()

Create a new empty HNSW index with default configuration

new(Options)

Create a new empty HNSW index with custom configuration Options: - m: Max connections per layer (default: 16) - m_max0: Max connections at layer 0 (default: 2*M) - ef_construction: Build-time ef (default: 200) - distance_fn: cosine | euclidean (default: cosine) - dimension: Vector dimension (default: 768) - quantization: scalar | turboquant | subspace_turboquant | none (default: scalar) - tq_bits: TurboQuant bits per component (default: 3, when quantization=turboquant or subspace_turboquant) - tq_seed: TurboQuant random seed (default: 42) - tq_m: Number of subspaces for subspace_turboquant (default: auto, based on dimension)

quantize(Vector)

Quantize a float vector to 8-bit signed integers (scalar quantization). Format: <<Scale:32/float-little, Components/binary>> This is the default/legacy function for backward compatibility.

quantize(Vector, Hnsw_config)

Quantize a vector using the method specified in config

scalar_quantize(Vector)

8-bit scalar quantization

search(Index, Query, K)

Search for k nearest neighbors

search(Hnsw_index, Query, K, Options)

Search for k nearest neighbors with options

serialize(Hnsw_index)

Serialize entire index to binary (includes full graph structure)

serialize_node(Hnsw_node)

Serialize a single node (for incremental persistence)

size(Hnsw_index)

Get index size

Types

hnsw_config/0

-type hnsw_config() ::
          #hnsw_config{m :: pos_integer(),
                       m_max0 :: pos_integer(),
                       ef_construction :: pos_integer(),
                       ml :: float() | undefined,
                       distance_fn :: cosine | euclidean,
                       quantization :: quantization_method(),
                       tq_state :: term() | undefined}.

hnsw_index/0

-type hnsw_index() ::
          #hnsw_index{entry_point :: binary() | undefined,
                      max_layer :: non_neg_integer(),
                      nodes :: #{binary() => hnsw_node()},
                      config :: hnsw_config(),
                      size :: non_neg_integer(),
                      dimension :: pos_integer()}.

hnsw_node/0

-type hnsw_node() ::
          #hnsw_node{id :: binary(),
                     vector :: quantized_vector(),
                     norm :: float(),
                     layer :: non_neg_integer(),
                     neighbors :: #{non_neg_integer() => [binary()]}}.

quantization_method/0

-type quantization_method() :: scalar | pq | turboquant | subspace_turboquant | none.

quantized_vector/0

-type quantized_vector() :: binary().

Functions

compute_norm(Vector)

-spec compute_norm([float()]) -> float().

Compute L2 norm of a float vector

cosine_distance(Vec1, Vec2)

-spec cosine_distance([float()], [float()]) -> float().

Cosine distance between two float vectors

cosine_similarity(Vec1, Vec2)

-spec cosine_similarity([float()], [float()]) -> float().

Cosine similarity between two float vectors

delete(Hnsw_index, Id)

-spec delete(hnsw_index(), binary()) -> hnsw_index().

Delete a node from the index

dequantize(_)

-spec dequantize(binary()) -> [float()].

Dequantize back to float vector (scalar quantization)

dequantize(Quantized, Hnsw_config)

-spec dequantize(binary(),
                 #hnsw_config{m :: pos_integer(),
                              m_max0 :: pos_integer(),
                              ef_construction :: pos_integer(),
                              ml :: float() | undefined,
                              distance_fn :: cosine | euclidean,
                              quantization :: quantization_method(),
                              tq_state :: term() | undefined}) ->
                    [float()].

Dequantize using the method specified in config

deserialize(Binary)

-spec deserialize(binary()) -> {ok, hnsw_index()} | {error, term()}.

Deserialize index from binary

deserialize_node(_)

-spec deserialize_node(binary()) -> {ok, map()} | {error, term()}.

Deserialize a single node

euclidean_distance(Vec1, Vec2)

-spec euclidean_distance([float()], [float()]) -> float().

Euclidean distance between two float vectors

get_node(Hnsw_index, Id)

-spec get_node(hnsw_index(), binary()) -> {ok, hnsw_node()} | not_found.

Get a node by ID

info(Hnsw_index)

-spec info(hnsw_index()) -> map().

Get index info

insert(Hnsw_index, Id, Vector)

-spec insert(hnsw_index(), binary(), [float()]) -> hnsw_index().

Insert a vector with given ID into the index Vector is quantized to 8-bit and norm is cached

new()

-spec new() -> hnsw_index().

Create a new empty HNSW index with default configuration

new(Options)

-spec new(map()) -> hnsw_index().

Create a new empty HNSW index with custom configuration Options: - m: Max connections per layer (default: 16) - m_max0: Max connections at layer 0 (default: 2*M) - ef_construction: Build-time ef (default: 200) - distance_fn: cosine | euclidean (default: cosine) - dimension: Vector dimension (default: 768) - quantization: scalar | turboquant | subspace_turboquant | none (default: scalar) - tq_bits: TurboQuant bits per component (default: 3, when quantization=turboquant or subspace_turboquant) - tq_seed: TurboQuant random seed (default: 42) - tq_m: Number of subspaces for subspace_turboquant (default: auto, based on dimension)

quantize(Vector)

-spec quantize([float()]) -> binary().

Quantize a float vector to 8-bit signed integers (scalar quantization). Format: <<Scale:32/float-little, Components/binary>> This is the default/legacy function for backward compatibility.

quantize(Vector, Hnsw_config)

-spec quantize([float()],
               #hnsw_config{m :: pos_integer(),
                            m_max0 :: pos_integer(),
                            ef_construction :: pos_integer(),
                            ml :: float() | undefined,
                            distance_fn :: cosine | euclidean,
                            quantization :: quantization_method(),
                            tq_state :: term() | undefined}) ->
                  binary().

Quantize a vector using the method specified in config

scalar_quantize(Vector)

-spec scalar_quantize([float()]) -> binary().

8-bit scalar quantization

search(Index, Query, K)

-spec search(hnsw_index(), [float()], pos_integer()) -> [{binary(), float()}].

Search for k nearest neighbors

search(Hnsw_index, Query, K, Options)

-spec search(hnsw_index(), [float()], pos_integer(), map()) -> [{binary(), float()}].

Search for k nearest neighbors with options

serialize(Hnsw_index)

-spec serialize(hnsw_index()) -> binary().

Serialize entire index to binary (includes full graph structure)

serialize_node(Hnsw_node)

-spec serialize_node(hnsw_node()) -> binary().

Serialize a single node (for incremental persistence)

size(Hnsw_index)

-spec size(hnsw_index()) -> non_neg_integer().

Get index size

barrel_vectordb_index behaviour

Vector index backend behaviour

Defines the interface that vector index backends must implement. Supports both pure Erlang HNSW and FAISS (via NIF) backends.

Available Backends

hnsw - Pure Erlang HNSW implementation (default)

faiss - Facebook FAISS via NIF binding (optional)

Usage

   %% Get the module for a backend
   Mod = barrel_vectordb_index:backend_module(hnsw),

   %% Create an index
   {ok, Index} = Mod:new(#{dimension => 128}),

   %% Insert vectors
   {ok, Index2} = Mod:insert(Index, <<"doc1">>, [0.1, 0.2, ...]),

   %% Search
   Results = Mod:search(Index2, [0.1, 0.2, ...], 10).

Summary

Types

backend/0

Available backend types. Note: hybrid backend has been removed in favor of pure DiskANN

index/0

Opaque index type. Implementation depends on the backend.

Callbacks

close/1

delete/2

deserialize/1

info/1

insert/3

new/1

search/3

search/4

serialize/1

size/1

Functions

backend_module(_)

Get the implementation module for a backend. Returns the module that implements the barrel_vectordb_index behaviour.

is_available(_)

Check if a backend is available. HNSW and diskann are always available. FAISS requires barrel_faiss NIF.

Types

backend/0

-type backend() :: hnsw | faiss | diskann.

Available backend types. Note: hybrid backend has been removed in favor of pure DiskANN

index/0

-type index() :: term().

Opaque index type. Implementation depends on the backend.

Callbacks

close/1

(optional)

-callback close(Index :: index()) -> ok.

delete/2

-callback delete(Index :: index(), Id :: binary()) -> {ok, index()} | {error, term()}.

deserialize/1

-callback deserialize(Binary :: binary()) -> {ok, index()} | {error, term()}.

info/1

-callback info(Index :: index()) -> map().

insert/3

-callback insert(Index :: index(), Id :: binary(), Vector :: [float()]) -> {ok, index()} | {error, term()}.

new/1

-callback new(Config :: map()) -> {ok, index()} | {error, term()}.

search/3

-callback search(Index :: index(), Query :: [float()], K :: pos_integer()) -> [{binary(), float()}].

search/4

-callback search(Index :: index(), Query :: [float()], K :: pos_integer(), Opts :: map()) ->
                    [{binary(), float()}].

serialize/1

-callback serialize(Index :: index()) -> binary().

size/1

-callback size(Index :: index()) -> non_neg_integer().

Functions

backend_module(_)

-spec backend_module(backend()) -> module().

Get the implementation module for a backend. Returns the module that implements the barrel_vectordb_index behaviour.

is_available(_)

-spec is_available(backend()) -> boolean().

Check if a backend is available. HNSW and diskann are always available. FAISS requires barrel_faiss NIF.

barrel_vectordb_index_faiss

FAISS backend wrapper for barrel_vectordb_index behaviour

Wraps barrel_faiss (Facebook FAISS NIF bindings) with: - ID mapping: binary IDs <-> sequential integer labels - Soft delete: deleted IDs filtered from search results - Metric normalization: cosine via normalized inner product - Combined serialization: FAISS binary + Erlang state

Delete Strategy

FAISS doesn't support deletion natively. This backend uses soft delete: deleted IDs are added to a set and filtered from search results. Use compact/1 to rebuild the index without deleted vectors.

Summary

Functions

close(Faiss_state)

Close and release FAISS index resources.

compact(Faiss_state)

Compact the index by rebuilding without deleted vectors. This reclaims space used by soft-deleted vectors.

delete(Faiss_state, Id)

Delete a vector by ID (soft delete).

deleted_count(Faiss_state)

Get the number of soft-deleted vectors.

deserialize(_)

Deserialize index from binary.

info(Faiss_state)

Get index information and statistics.

insert(Faiss_state, Id, Vector)

Insert a vector with the given ID.

new(Config)

Create a new FAISS index. Config options: - dimension: Vector dimension (required) - distance_fn: cosine | euclidean (default: cosine) - index_type: FAISS factory string (default: "HNSW32")

search(Faiss_state, Query, K)

Search for K nearest neighbors.

search(Faiss_state, Query, K, Opts)

Search for K nearest neighbors with options.

serialize(Faiss_state)

Serialize index to binary. Format: <<Version, FAISSLen, FAISSBin, StateLen, StateBin>>

size(Faiss_state)

Get the number of active (non-deleted) vectors.

Functions

close(Faiss_state)

-spec close(#faiss_state{index :: reference(),
                         dimension :: pos_integer(),
                         metric :: l2 | inner_product,
                         distance_fn :: cosine | euclidean,
                         id_to_label :: #{binary() => non_neg_integer()},
                         label_to_id :: #{non_neg_integer() => binary()},
                         vectors :: #{binary() => [float()]},
                         next_label :: non_neg_integer(),
                         deleted_ids :: sets:set(binary()),
                         config :: map()}) ->
               ok.

Close and release FAISS index resources.

compact(Faiss_state)

-spec compact(#faiss_state{index :: reference(),
                           dimension :: pos_integer(),
                           metric :: l2 | inner_product,
                           distance_fn :: cosine | euclidean,
                           id_to_label :: #{binary() => non_neg_integer()},
                           label_to_id :: #{non_neg_integer() => binary()},
                           vectors :: #{binary() => [float()]},
                           next_label :: non_neg_integer(),
                           deleted_ids :: sets:set(binary()),
                           config :: map()}) ->
                 {ok,
                  #faiss_state{index :: reference(),
                               dimension :: pos_integer(),
                               metric :: l2 | inner_product,
                               distance_fn :: cosine | euclidean,
                               id_to_label :: #{binary() => non_neg_integer()},
                               label_to_id :: #{non_neg_integer() => binary()},
                               vectors :: #{binary() => [float()]},
                               next_label :: non_neg_integer(),
                               deleted_ids :: sets:set(binary()),
                               config :: map()}} |
                 {error, term()}.

Compact the index by rebuilding without deleted vectors. This reclaims space used by soft-deleted vectors.

delete(Faiss_state, Id)

-spec delete(#faiss_state{index :: reference(),
                          dimension :: pos_integer(),
                          metric :: l2 | inner_product,
                          distance_fn :: cosine | euclidean,
                          id_to_label :: #{binary() => non_neg_integer()},
                          label_to_id :: #{non_neg_integer() => binary()},
                          vectors :: #{binary() => [float()]},
                          next_label :: non_neg_integer(),
                          deleted_ids :: sets:set(binary()),
                          config :: map()},
             binary()) ->
                {ok,
                 #faiss_state{index :: reference(),
                              dimension :: pos_integer(),
                              metric :: l2 | inner_product,
                              distance_fn :: cosine | euclidean,
                              id_to_label :: #{binary() => non_neg_integer()},
                              label_to_id :: #{non_neg_integer() => binary()},
                              vectors :: #{binary() => [float()]},
                              next_label :: non_neg_integer(),
                              deleted_ids :: sets:set(binary()),
                              config :: map()}} |
                {error, term()}.

Delete a vector by ID (soft delete).

deleted_count(Faiss_state)

-spec deleted_count(#faiss_state{index :: reference(),
                                 dimension :: pos_integer(),
                                 metric :: l2 | inner_product,
                                 distance_fn :: cosine | euclidean,
                                 id_to_label :: #{binary() => non_neg_integer()},
                                 label_to_id :: #{non_neg_integer() => binary()},
                                 vectors :: #{binary() => [float()]},
                                 next_label :: non_neg_integer(),
                                 deleted_ids :: sets:set(binary()),
                                 config :: map()}) ->
                       non_neg_integer().

Get the number of soft-deleted vectors.

deserialize(_)

-spec deserialize(binary()) ->
                     {ok,
                      #faiss_state{index :: reference(),
                                   dimension :: pos_integer(),
                                   metric :: l2 | inner_product,
                                   distance_fn :: cosine | euclidean,
                                   id_to_label :: #{binary() => non_neg_integer()},
                                   label_to_id :: #{non_neg_integer() => binary()},
                                   vectors :: #{binary() => [float()]},
                                   next_label :: non_neg_integer(),
                                   deleted_ids :: sets:set(binary()),
                                   config :: map()}} |
                     {error, term()}.

Deserialize index from binary.

info(Faiss_state)

-spec info(#faiss_state{index :: reference(),
                        dimension :: pos_integer(),
                        metric :: l2 | inner_product,
                        distance_fn :: cosine | euclidean,
                        id_to_label :: #{binary() => non_neg_integer()},
                        label_to_id :: #{non_neg_integer() => binary()},
                        vectors :: #{binary() => [float()]},
                        next_label :: non_neg_integer(),
                        deleted_ids :: sets:set(binary()),
                        config :: map()}) ->
              map().

Get index information and statistics.

insert(Faiss_state, Id, Vector)

-spec insert(#faiss_state{index :: reference(),
                          dimension :: pos_integer(),
                          metric :: l2 | inner_product,
                          distance_fn :: cosine | euclidean,
                          id_to_label :: #{binary() => non_neg_integer()},
                          label_to_id :: #{non_neg_integer() => binary()},
                          vectors :: #{binary() => [float()]},
                          next_label :: non_neg_integer(),
                          deleted_ids :: sets:set(binary()),
                          config :: map()},
             binary(),
             [float()]) ->
                {ok,
                 #faiss_state{index :: reference(),
                              dimension :: pos_integer(),
                              metric :: l2 | inner_product,
                              distance_fn :: cosine | euclidean,
                              id_to_label :: #{binary() => non_neg_integer()},
                              label_to_id :: #{non_neg_integer() => binary()},
                              vectors :: #{binary() => [float()]},
                              next_label :: non_neg_integer(),
                              deleted_ids :: sets:set(binary()),
                              config :: map()}} |
                {error, term()}.

Insert a vector with the given ID.

new(Config)

-spec new(map()) ->
             {ok,
              #faiss_state{index :: reference(),
                           dimension :: pos_integer(),
                           metric :: l2 | inner_product,
                           distance_fn :: cosine | euclidean,
                           id_to_label :: #{binary() => non_neg_integer()},
                           label_to_id :: #{non_neg_integer() => binary()},
                           vectors :: #{binary() => [float()]},
                           next_label :: non_neg_integer(),
                           deleted_ids :: sets:set(binary()),
                           config :: map()}} |
             {error, term()}.

Create a new FAISS index. Config options: - dimension: Vector dimension (required) - distance_fn: cosine | euclidean (default: cosine) - index_type: FAISS factory string (default: "HNSW32")

search(Faiss_state, Query, K)

-spec search(#faiss_state{index :: reference(),
                          dimension :: pos_integer(),
                          metric :: l2 | inner_product,
                          distance_fn :: cosine | euclidean,
                          id_to_label :: #{binary() => non_neg_integer()},
                          label_to_id :: #{non_neg_integer() => binary()},
                          vectors :: #{binary() => [float()]},
                          next_label :: non_neg_integer(),
                          deleted_ids :: sets:set(binary()),
                          config :: map()},
             [float()],
             pos_integer()) ->
                [{binary(), float()}].

Search for K nearest neighbors.

search(Faiss_state, Query, K, Opts)

-spec search(#faiss_state{index :: reference(),
                          dimension :: pos_integer(),
                          metric :: l2 | inner_product,
                          distance_fn :: cosine | euclidean,
                          id_to_label :: #{binary() => non_neg_integer()},
                          label_to_id :: #{non_neg_integer() => binary()},
                          vectors :: #{binary() => [float()]},
                          next_label :: non_neg_integer(),
                          deleted_ids :: sets:set(binary()),
                          config :: map()},
             [float()],
             pos_integer(),
             map()) ->
                [{binary(), float()}].

Search for K nearest neighbors with options.

serialize(Faiss_state)

-spec serialize(#faiss_state{index :: reference(),
                             dimension :: pos_integer(),
                             metric :: l2 | inner_product,
                             distance_fn :: cosine | euclidean,
                             id_to_label :: #{binary() => non_neg_integer()},
                             label_to_id :: #{non_neg_integer() => binary()},
                             vectors :: #{binary() => [float()]},
                             next_label :: non_neg_integer(),
                             deleted_ids :: sets:set(binary()),
                             config :: map()}) ->
                   binary().

Serialize index to binary. Format: <<Version, FAISSLen, FAISSBin, StateLen, StateBin>>

size(Faiss_state)

-spec size(#faiss_state{index :: reference(),
                        dimension :: pos_integer(),
                        metric :: l2 | inner_product,
                        distance_fn :: cosine | euclidean,
                        id_to_label :: #{binary() => non_neg_integer()},
                        label_to_id :: #{non_neg_integer() => binary()},
                        vectors :: #{binary() => [float()]},
                        next_label :: non_neg_integer(),
                        deleted_ids :: sets:set(binary()),
                        config :: map()}) ->
              non_neg_integer().

Get the number of active (non-deleted) vectors.

barrel_vectordb_index_hnsw

HNSW backend wrapper for barrel_vectordb_index behaviour

Thin wrapper around barrel_vectordb_hnsw that adapts its API to the barrel_vectordb_index behaviour specification.

Summary

Types

hnsw_config/0

hnsw_index/0

hnsw_node/0

quantization_method/0

quantized_vector/0

Functions

delete(Index, Id)

Delete a vector by ID.

deserialize(Binary)

Deserialize index from binary.

info(Index)

Get index information and statistics.

insert(Index, Id, Vector)

Insert a vector with the given ID.

new(Config)

Create a new HNSW index.

search(Index, Query, K)

Search for K nearest neighbors.

search(Index, Query, K, Opts)

Search for K nearest neighbors with options.

serialize(Index)

Serialize index to binary.

size(Index)

Get the number of vectors in the index.

Types

hnsw_config/0

-type hnsw_config() ::
          #hnsw_config{m :: pos_integer(),
                       m_max0 :: pos_integer(),
                       ef_construction :: pos_integer(),
                       ml :: float() | undefined,
                       distance_fn :: cosine | euclidean,
                       quantization :: quantization_method(),
                       tq_state :: term() | undefined}.

hnsw_index/0

-type hnsw_index() ::
          #hnsw_index{entry_point :: binary() | undefined,
                      max_layer :: non_neg_integer(),
                      nodes :: #{binary() => hnsw_node()},
                      config :: hnsw_config(),
                      size :: non_neg_integer(),
                      dimension :: pos_integer()}.

hnsw_node/0

-type hnsw_node() ::
          #hnsw_node{id :: binary(),
                     vector :: quantized_vector(),
                     norm :: float(),
                     layer :: non_neg_integer(),
                     neighbors :: #{non_neg_integer() => [binary()]}}.

quantization_method/0

-type quantization_method() :: scalar | pq | turboquant | subspace_turboquant | none.

quantized_vector/0

-type quantized_vector() :: binary().

Functions

delete(Index, Id)

-spec delete(hnsw_index(), binary()) -> {ok, hnsw_index()} | {error, term()}.

Delete a vector by ID.

deserialize(Binary)

-spec deserialize(binary()) -> {ok, hnsw_index()} | {error, term()}.

Deserialize index from binary.

info(Index)

-spec info(hnsw_index()) -> map().

Get index information and statistics.

insert(Index, Id, Vector)

-spec insert(hnsw_index(), binary(), [float()]) -> {ok, hnsw_index()} | {error, term()}.

Insert a vector with the given ID.

new(Config)

-spec new(map()) -> {ok, hnsw_index()} | {error, term()}.

Create a new HNSW index.

search(Index, Query, K)

-spec search(hnsw_index(), [float()], pos_integer()) -> [{binary(), float()}].

Search for K nearest neighbors.

search(Index, Query, K, Opts)

-spec search(hnsw_index(), [float()], pos_integer(), map()) -> [{binary(), float()}].

Search for K nearest neighbors with options.

serialize(Index)

-spec serialize(hnsw_index()) -> binary().

Serialize index to binary.

size(Index)

-spec size(hnsw_index()) -> non_neg_integer().

Get the number of vectors in the index.

barrel_vectordb_nif

Unified NIF wrapper for barrel_vectordb SIMD operations.

Provides access to all NIF-accelerated functions including: - Distance computations (dot product, cosine, euclidean) - TurboQuant ADC distance - Subspace-TurboQuant ADC distance - Binary/list conversions

Summary

Functions

cosine_distance(Bin1, Bin2)

Compute cosine distance: 1 - cos(a, b).

cosine_distance_normalized(Bin1, Bin2)

Compute cosine distance for pre-normalized vectors.

dot_product(Bin1, Bin2)

Compute dot product of two binary vectors.

euclidean_distance(Bin1, Bin2)

Compute Euclidean distance.

from_binary(Bin)

Convert binary vector to list of floats.

simd_info()

Return SIMD backend info.

to_binary(List)

Convert list of floats to binary vector (32-bit floats).

tq_adc_distance(Tables, Code, Bits)

Compute TurboQuant ADC distance.

tq_batch_adc_distance(Tables, Codes, Bits)

Compute TurboQuant ADC distance for multiple codes.

tqs_adc_distance(Tables, Code, Bits, M)

Compute Subspace-TurboQuant ADC distance.

tqs_batch_adc_distance(Tables, Codes, Bits, M)

Compute Subspace-TurboQuant ADC distance for multiple codes.

Functions

cosine_distance(Bin1, Bin2)

-spec cosine_distance(binary(), binary()) -> float().

Compute cosine distance: 1 - cos(a, b).

cosine_distance_normalized(Bin1, Bin2)

-spec cosine_distance_normalized(binary(), binary()) -> float().

Compute cosine distance for pre-normalized vectors.

dot_product(Bin1, Bin2)

-spec dot_product(binary(), binary()) -> float().

Compute dot product of two binary vectors.

euclidean_distance(Bin1, Bin2)

-spec euclidean_distance(binary(), binary()) -> float().

Compute Euclidean distance.

from_binary(Bin)

-spec from_binary(binary()) -> [float()].

Convert binary vector to list of floats.

simd_info()

-spec simd_info() -> avx2 | neon | scalar.

Return SIMD backend info.

to_binary(List)

-spec to_binary([float()]) -> binary().

Convert list of floats to binary vector (32-bit floats).

tq_adc_distance(Tables, Code, Bits)

-spec tq_adc_distance(binary(), binary(), 2..4) -> float().

Compute TurboQuant ADC distance.

tq_batch_adc_distance(Tables, Codes, Bits)

-spec tq_batch_adc_distance(binary(), [binary()], 2..4) -> [float()].

Compute TurboQuant ADC distance for multiple codes.

tqs_adc_distance(Tables, Code, Bits, M)

-spec tqs_adc_distance(binary(), binary(), 2..4, pos_integer()) -> float().

Compute Subspace-TurboQuant ADC distance.

tqs_batch_adc_distance(Tables, Codes, Bits, M)

-spec tqs_batch_adc_distance(binary(), [binary()], 2..4, pos_integer()) -> [float()].

Compute Subspace-TurboQuant ADC distance for multiple codes.

barrel_vectordb_pq

Product Quantization for DiskANN

Product Quantization (PQ) compresses vectors by: 1. Splitting D-dimensional vector into M subspaces 2. Training K centroids per subspace (k-means) 3. Encoding vector as M bytes (indices into codebooks)

Distance computation uses precomputed lookup tables: - For query q, precompute distance from q's subvectors to all centroids - Distance to any encoded vector = sum of M table lookups

Memory: M bytes per vector (vs D*4 for float32) With M=8, K=256: ~32x compression for 256-dim vectors

Summary

Types

distance_tables/0

M*K float32 values

pq_code/0

M bytes

pq_config/0

Functions

batch_encode(Config, Vectors)

Batch encode multiple vectors

decode(Pq_config, Code)

Decode PQ code back to approximate vector

distance(Tables, Code)

Compute asymmetric distance using precomputed tables This is the fast O(M) lookup vs O(D) for full distance Tables size must be M*K*4 where M = byte_size(Code)

encode(Pq_config, Vec)

Encode a vector to M bytes using trained codebooks

info(Pq_config)

Get PQ configuration info

is_trained(Pq_config)

Check if PQ is trained

kmeans(Vectors, K, MaxIter)

new(Options)

Create a new PQ configuration Options: m - number of subquantizers (default: 8) k - centroids per subquantizer (default: 256, max: 256) dimension - vector dimension (required, must be divisible by m)

precompute_tables(Pq_config, Query)

Precompute distance tables for a query vector Returns M*K float32 values: distance from each query subvector to each centroid

split_subvectors(Vec, M, SubDim)

train(Pq_config, Vectors)

Train codebooks using k-means clustering Vectors should be a list of float lists, all same dimension

Types

distance_tables/0

-type distance_tables() :: binary().

M*K float32 values

pq_code/0

-type pq_code() :: binary().

M bytes

pq_config/0

-type pq_config() ::
          #pq_config{m :: pos_integer(),
                     k :: pos_integer(),
                     dimension :: pos_integer(),
                     subvector_dim :: pos_integer(),
                     codebooks :: [binary()] | undefined,
                     trained :: boolean()}.

Functions

batch_encode(Config, Vectors)

-spec batch_encode(pq_config(), [[float()]]) -> [pq_code()].

Batch encode multiple vectors

decode(Pq_config, Code)

-spec decode(pq_config(), pq_code()) -> [float()].

Decode PQ code back to approximate vector

distance(Tables, Code)

-spec distance(distance_tables(), pq_code()) -> float().

Compute asymmetric distance using precomputed tables This is the fast O(M) lookup vs O(D) for full distance Tables size must be M*K*4 where M = byte_size(Code)

encode(Pq_config, Vec)

-spec encode(pq_config(), [float()]) -> pq_code().

Encode a vector to M bytes using trained codebooks

info(Pq_config)

-spec info(pq_config()) -> map().

Get PQ configuration info

is_trained(Pq_config)

-spec is_trained(pq_config()) -> boolean().

Check if PQ is trained

kmeans(Vectors, K, MaxIter)

new(Options)

-spec new(map()) -> {ok, pq_config()} | {error, term()}.

Create a new PQ configuration Options: m - number of subquantizers (default: 8) k - centroids per subquantizer (default: 256, max: 256) dimension - vector dimension (required, must be divisible by m)

precompute_tables(Pq_config, Query)

-spec precompute_tables(pq_config(), [float()]) -> distance_tables().

Precompute distance tables for a query vector Returns M*K float32 values: distance from each query subvector to each centroid

split_subvectors(Vec, M, SubDim)

train(Pq_config, Vectors)

-spec train(pq_config(), [[float()]]) -> {ok, pq_config()} | {error, term()}.

Train codebooks using k-means clustering Vectors should be a list of float lists, all same dimension

barrel_vectordb_registry

Local name registry for vector stores and their companions.

Names are arbitrary terms (in practice {vstore, NameBin} and friends), so dynamically named ephemeral stores never grow the atom table the way {local, Atom} registration does. Implements the {via, Module, Name} callbacks over a public ETS table; the owner process monitors registered pids and removes dead entries, so a crashed store never leaves a stale registration behind.

Summary

Functions

ensure()

Start the registry if it is not running. Covers callers that start stores without the barrel_vectordb application (eunit, embedded use); under the application the supervisor owns it.

handle_call(_, From, Mons)

handle_cast(Msg, Mons)

handle_info(Info, Mons)

init(_)

register_name(Name, Pid)

via callback: register Name to Pid. yes on success, no if the name is taken by a live process.

send(Name, Msg)

via callback.

start_link()

unregister_name(Name)

via callback.

whereis_name(Name)

via callback: resolve Name to a live pid or undefined.

Functions

ensure()

-spec ensure() -> ok.

Start the registry if it is not running. Covers callers that start stores without the barrel_vectordb application (eunit, embedded use); under the application the supervisor owns it.

handle_call(_, From, Mons)

handle_cast(Msg, Mons)

handle_info(Info, Mons)

init(_)

register_name(Name, Pid)

-spec register_name(term(), pid()) -> yes | no.

via callback: register Name to Pid. yes on success, no if the name is taken by a live process.

send(Name, Msg)

-spec send(term(), term()) -> pid().

via callback.

start_link()

-spec start_link() -> {ok, pid()} | {error, term()}.

unregister_name(Name)

-spec unregister_name(term()) -> ok.

via callback.

whereis_name(Name)

-spec whereis_name(term()) -> pid() | undefined.

via callback: resolve Name to a live pid or undefined.

barrel_vectordb_server

Per-store gen_batch_server managing RocksDB, vector index, and embeddings

Each store runs as a separate gen_batch_server registered under its name. Handles all document operations, search, and embedding coordination.

Uses gen_batch_server to automatically batch concurrent write operations into single atomic RocksDB WriteBatch operations, improving throughput under concurrent load.

Supports pluggable vector index backends: - hnsw: Pure Erlang HNSW implementation (default) - faiss: Facebook FAISS via NIF binding (optional, requires barrel_faiss)

Summary

Types

store_ref/0

Functions

add(Store, Id, Text, Metadata)

Add document with auto-embedding.

add_batch(Store, Docs)

Add multiple documents.

add_index_only(Store, Id, Text, Vector)

Index a vector without storing text/metadata (see barrel_vectordb).

add_index_only_batch(Store, Entries)

Index multiple vectors without storing text/metadata.

add_vector(Store, Id, Text, Metadata, Vector)

Add document with explicit vector.

add_vector_batch(Store, Docs)

Add multiple documents with pre-computed vectors (bulk insert).

bm25_compact(Store)

Trigger BM25 index compaction (disk backend only).

bm25_info(Store)

Get BM25 index information.

checkpoint(Store)

Checkpoint HNSW index to disk.

count(Store)

Get document count.

delete(Store, Id)

Delete document.

embed(Store, Text)

Embed single text.

embed_batch(Store, Texts)

Embed multiple texts.

embedder_info(Store)

Get embedder information.

get(Store, Id)

Get document by ID.

get_db_path(Store)

The store's storage path (RocksDB dir; bm25/diskann nest under it).

handle_batch(Ops, State)

Handle a batch of operations. Partitions operations into reads (processed immediately) and writes (batched atomically).

init(_)

peek(Store, Limit)

Peek at documents (sample without search).

search(Store, Query, Opts)

Search with text query.

search_bm25(Store, Query, Opts)

Search with BM25 text query. Opts: #{k => integer()}

search_hybrid(Store, Query, Opts)

Hybrid search combining BM25 and vector search. Opts: #{k => integer(), bm25_weight => float(), vector_weight => float(), fusion => rrf | linear}

search_vector(Store, Vector, Opts)

Search with vector query.

start_link(Name, Config)

Start a named store. Config options: - db_path: RocksDB storage path - dimension: Vector dimension - hnsw: HNSW index configuration - batch: gen_batch_server options (max_batch_size, min_batch_size)

stats(Store)

Get store statistics.

stop(Store)

Stop a store.

terminate(Reason, State)

update(Store, Id, Text, Metadata)

Update document metadata (re-embeds the text).

upsert(Store, Id, Text, Metadata)

Insert or update document.

Types

store_ref/0

-type store_ref() :: atom() | binary() | pid().

Functions

add(Store, Id, Text, Metadata)

-spec add(store_ref(), binary(), binary(), map()) -> ok | {error, term()}.

Add document with auto-embedding.

add_batch(Store, Docs)

-spec add_batch(store_ref(), [{binary(), binary(), map()}]) ->
                   {ok, #{inserted := non_neg_integer()}} | {error, term()}.

Add multiple documents.

add_index_only(Store, Id, Text, Vector)

-spec add_index_only(store_ref(), binary(), binary(), [float()]) -> ok | {error, term()}.

Index a vector without storing text/metadata (see barrel_vectordb).

add_index_only_batch(Store, Entries)

-spec add_index_only_batch(store_ref(), [{binary(), binary(), [float()]}]) ->
                              {ok, #{inserted := non_neg_integer()}} | {error, term()}.

Index multiple vectors without storing text/metadata.

add_vector(Store, Id, Text, Metadata, Vector)

-spec add_vector(store_ref(), binary(), binary(), map(), [float()]) -> ok | {error, term()}.

Add document with explicit vector.

add_vector_batch(Store, Docs)

-spec add_vector_batch(store_ref(), [{binary(), binary(), map(), [float()]}]) ->
                          {ok, #{inserted := non_neg_integer()}} | {error, term()}.

Add multiple documents with pre-computed vectors (bulk insert).

bm25_compact(Store)

-spec bm25_compact(store_ref()) -> ok | {error, term()}.

Trigger BM25 index compaction (disk backend only).

bm25_info(Store)

-spec bm25_info(store_ref()) -> {ok, map()} | {error, bm25_not_enabled}.

Get BM25 index information.

checkpoint(Store)

-spec checkpoint(store_ref()) -> ok.

Checkpoint HNSW index to disk.

count(Store)

-spec count(store_ref()) -> non_neg_integer().

Get document count.

delete(Store, Id)

-spec delete(store_ref(), binary()) -> ok | {error, term()}.

Delete document.

embed(Store, Text)

-spec embed(store_ref(), binary()) -> {ok, [float()]} | {error, term()}.

Embed single text.

embed_batch(Store, Texts)

-spec embed_batch(store_ref(), [binary()]) -> {ok, [[float()]]} | {error, term()}.

Embed multiple texts.

embedder_info(Store)

-spec embedder_info(store_ref()) -> {ok, map()}.

Get embedder information.

get(Store, Id)

-spec get(store_ref(), binary()) -> {ok, map()} | not_found | {error, term()}.

Get document by ID.

get_db_path(Store)

-spec get_db_path(store_ref()) -> {ok, string()}.

The store's storage path (RocksDB dir; bm25/diskann nest under it).

handle_batch(Ops, State)

Handle a batch of operations. Partitions operations into reads (processed immediately) and writes (batched atomically).

init(_)

peek(Store, Limit)

-spec peek(store_ref(), pos_integer()) -> {ok, [map()]}.

Peek at documents (sample without search).

search(Store, Query, Opts)

-spec search(store_ref(), binary(), map()) -> {ok, [map()]} | {error, term()}.

Search with text query.

search_bm25(Store, Query, Opts)

-spec search_bm25(store_ref(), binary(), map()) -> {ok, [{binary(), float()}]} | {error, term()}.

Search with BM25 text query. Opts: #{k => integer()}

search_hybrid(Store, Query, Opts)

-spec search_hybrid(store_ref(), binary(), map()) -> {ok, [map()]} | {error, term()}.

Hybrid search combining BM25 and vector search. Opts: #{k => integer(), bm25_weight => float(), vector_weight => float(), fusion => rrf | linear}

search_vector(Store, Vector, Opts)

-spec search_vector(store_ref(), [float()], map()) -> {ok, [map()]} | {error, term()}.

Search with vector query.

start_link(Name, Config)

-spec start_link(atom() | binary(), map()) -> {ok, pid()} | {error, term()}.

Start a named store. Config options: - db_path: RocksDB storage path - dimension: Vector dimension - hnsw: HNSW index configuration - batch: gen_batch_server options (max_batch_size, min_batch_size)

Default batch settings optimized for vector DB workloads: - min_batch_size: 4 (responsive for single inserts, batches concurrent ones) - max_batch_size: 256 (reasonable upper bound for memory/latency)

stats(Store)

-spec stats(store_ref()) -> {ok, map()}.

Get store statistics.

stop(Store)

-spec stop(store_ref()) -> ok.

Stop a store.

terminate(Reason, State)

update(Store, Id, Text, Metadata)

-spec update(store_ref(), binary(), binary(), map()) -> ok | not_found | {error, term()}.

Update document metadata (re-embeds the text).

upsert(Store, Id, Text, Metadata)

-spec upsert(store_ref(), binary(), binary(), map()) -> ok | {error, term()}.

Insert or update document.

barrel_vectordb_store

Vector store gen_server managing RocksDB and HNSW index

Uses RocksDB column families: - vectors: chunk_id → binary float32 array - metadata: chunk_id → term_to_binary(map) - text: chunk_id → UTF-8 binary - hnsw_graph: node_id → serialized HNSW node

Summary

Types

metadata/0

search_result/0

vector/0

Functions

add(ChunkId, Text, Metadata)

Add a chunk with auto-embedding

add(ChunkId, Text, Metadata, Vector)

Add a chunk with explicit vector

count()

Get chunk count

delete(ChunkId)

Delete a chunk

get(ChunkId)

Get a chunk by ID

get_diskann_db()

Get DiskANN ID mapping database handles Returns {ok, {Db, CfFwd, CfRev}} | {error, not_started}

handle_call(Request, From, State)

handle_cast(Msg, State)

handle_info(Info, State)

info()

Get store info

init(Config)

rebuild_index()

Rebuild HNSW index from stored vectors

search(Query, K)

Search for similar chunks

search(Query, K, Options)

Search for similar chunks with options

start_link(Config)

Start the vector store

stop()

Stop the vector store

terminate(Reason, State)

Types

metadata/0

-type metadata() ::
          #{file => binary(),
            start_line => pos_integer(),
            end_line => pos_integer(),
            type => atom(),
            atom() => term()}.

search_result/0

-type search_result() ::
          #{id := binary(),
            text := binary(),
            metadata := metadata(),
            score := float(),
            vector => vector()}.

vector/0

-type vector() :: [float()] | binary().

Functions

add(ChunkId, Text, Metadata)

-spec add(binary(), binary(), metadata()) -> ok | {error, term()}.

Add a chunk with auto-embedding

add(ChunkId, Text, Metadata, Vector)

-spec add(binary(), binary(), metadata(), [float()]) -> ok | {error, term()}.

Add a chunk with explicit vector

count()

-spec count() -> non_neg_integer().

Get chunk count

delete(ChunkId)

-spec delete(binary()) -> ok | {error, term()}.

Delete a chunk

get(ChunkId)

-spec get(binary()) -> {ok, map()} | not_found | {error, term()}.

Get a chunk by ID

get_diskann_db()

-spec get_diskann_db() ->
                        {ok, {rocksdb:db_handle(), rocksdb:cf_handle(), rocksdb:cf_handle()}} |
                        {error, term()}.

Get DiskANN ID mapping database handles Returns {ok, {Db, CfFwd, CfRev}} | {error, not_started}

handle_call(Request, From, State)

handle_cast(Msg, State)

handle_info(Info, State)

info()

-spec info() -> map().

Get store info

init(Config)

rebuild_index()

-spec rebuild_index() -> ok | {error, term()}.

Rebuild HNSW index from stored vectors

search(Query, K)

-spec search(binary() | [float()], pos_integer()) -> {ok, [search_result()]} | {error, term()}.

Search for similar chunks

search(Query, K, Options)

-spec search(binary() | [float()], pos_integer(), map()) -> {ok, [search_result()]} | {error, term()}.

Search for similar chunks with options

start_link(Config)

-spec start_link(map()) -> {ok, pid()} | {error, term()}.

Start the vector store

stop()

-spec stop() -> ok.

Stop the vector store

terminate(Reason, State)

barrel_vectordb_sup

barrel_vectordb top level supervisor

This supervisor is the root of the barrel_vectordb OTP application. barrel_vectordb is an embedded library: individual stores are started on demand by the embedding application via barrel_vectordb:start_link/1. Its only static child is the name registry stores register in (barrel_vectordb_registry), which maps binary store names to pids without minting atoms.

Summary

Functions

start_link()

Start the barrel_vectordb supervisor.

Functions

start_link()

-spec start_link() -> {ok, pid()} | {error, term()}.

Start the barrel_vectordb supervisor.

barrel_vectordb_turboquant

TurboQuant: Data-oblivious 3-bit vector quantization

TurboQuant provides efficient vector compression without training. Based on Google Research's algorithm combining:

1. PolarQuant: Random rotation + polar coordinate conversion 2. QJL: 1-bit Johnson-Lindenstrauss error correction

Key advantages over Product Quantization (PQ): - No training required (data-oblivious) - Deterministic with seed (reproducible results) - ~8x compression for 768-dim vectors (vs 4x for 8-bit scalar) - Only 1-3% recall loss vs float32

Storage format (D=768, 3-bit): Header: 4 bytes (version, bits, dimension flags) PolarQuant: ceil(D*bits/8) bytes QJL: ceil(D/8) bytes Total: ~388 bytes vs 3072 bytes float32

Summary

Types

distance_tables/0

tq_code/0

tq_config/0

Functions

apply_inverse_rotation(RotMatBin, Vector)

apply_rotation(RotMatBin, Vector)

batch_distance_nif(Tables, Codes)

Compute ADC distance for multiple codes using SIMD-accelerated NIF. Amortizes NIF call overhead for batch operations.

batch_encode(Config, Vectors)

Batch encode multiple vectors

compute_qjl_signs(QJLMatrix, Vector)

decode(Tq_config, _)

Decode TurboQuant code back to approximate vector

distance(Tables, _)

Compute asymmetric distance using precomputed tables Full ADC with correct polar distance formula: d^2 = r_q^2 + r_d^2 - 2*r_q*r_d*cos(theta_diff) = QRSq + r_d^2 - CosTerm * r_d

distance_nif(Tables, Code)

Compute ADC distance using SIMD-accelerated NIF. This is the optimized version that should be used for production workloads.

encode(Tq_config, Vector)

Encode a vector using TurboQuant Returns compact binary representation

from_polar(PolarVec, NumPairs)

generate_rotation_matrix(Dim, Seed)

info(Tq_config)

Get configuration info

new(Options)

Create a new TurboQuant configuration (no training needed) Options: bits - bits per component (default: 3, range: 2-4) qjl_bits - QJL error correction bits (default: 1) dimension - vector dimension (required, must be even) seed - random seed for rotation matrix (default: 42) qjl_iterations - QJL correction iterations (default: 5) qjl_learning_rate - QJL gradient learning rate (default: 0.1)

precompute_tables(Tq_config, Query)

Precompute distance lookup tables for a query vector This enables fast asymmetric distance computation (ADC)

to_polar(Vector, Levels)

Types

distance_tables/0

-type distance_tables() :: binary().

tq_code/0

-type tq_code() :: binary().

tq_config/0

-type tq_config() ::
          #tq_config{bits :: 2..4,
                     qjl_bits :: pos_integer(),
                     dimension :: pos_integer(),
                     rotation_seed :: integer(),
                     rotation_matrix :: binary(),
                     qjl_matrix :: binary(),
                     qjl_dim :: pos_integer(),
                     angle_levels :: [float()],
                     qjl_iterations :: non_neg_integer(),
                     qjl_learning_rate :: float()}.

Functions

apply_inverse_rotation(RotMatBin, Vector)

apply_rotation(RotMatBin, Vector)

batch_distance_nif(Tables, Codes)

-spec batch_distance_nif(distance_tables(), [tq_code()]) -> [float()].

Compute ADC distance for multiple codes using SIMD-accelerated NIF. Amortizes NIF call overhead for batch operations.

batch_encode(Config, Vectors)

-spec batch_encode(tq_config(), [[float()]]) -> [tq_code()].

Batch encode multiple vectors

compute_qjl_signs(QJLMatrix, Vector)

decode(Tq_config, _)

-spec decode(tq_config(), tq_code()) -> [float()].

Decode TurboQuant code back to approximate vector

distance(Tables, _)

-spec distance(distance_tables(), tq_code()) -> float().

Compute asymmetric distance using precomputed tables Full ADC with correct polar distance formula: d^2 = r_q^2 + r_d^2 - 2*r_q*r_d*cos(theta_diff) = QRSq + r_d^2 - CosTerm * r_d

Performance note: This ADC computation can be moved to C/NIF for better performance if needed, similar to barrel_vectordb_hnsw distance functions. The tight loop over pairs with table lookups would benefit from SIMD.

distance_nif(Tables, Code)

-spec distance_nif(distance_tables(), tq_code()) -> float().

Compute ADC distance using SIMD-accelerated NIF. This is the optimized version that should be used for production workloads.

encode(Tq_config, Vector)

-spec encode(tq_config(), [float()]) -> tq_code().

Encode a vector using TurboQuant Returns compact binary representation

from_polar(PolarVec, NumPairs)

generate_rotation_matrix(Dim, Seed)

info(Tq_config)

-spec info(tq_config()) -> map().

Get configuration info

new(Options)

-spec new(map()) -> {ok, tq_config()} | {error, term()}.

Create a new TurboQuant configuration (no training needed) Options: bits - bits per component (default: 3, range: 2-4) qjl_bits - QJL error correction bits (default: 1) dimension - vector dimension (required, must be even) seed - random seed for rotation matrix (default: 42) qjl_iterations - QJL correction iterations (default: 5) qjl_learning_rate - QJL gradient learning rate (default: 0.1)

precompute_tables(Tq_config, Query)

-spec precompute_tables(tq_config(), [float()]) -> distance_tables().

Precompute distance lookup tables for a query vector This enables fast asymmetric distance computation (ADC)

Full ADC table format per pair: QRSq (32-bit float) + NumLevels * CosTerm (32-bit floats) where CosTerm_i = 2 * r_q * cos(theta_q - angle_center_i)

Distance formula: d^2 = r_q^2 + r_d^2 - 2*r_q*r_d*cos(theta_diff) = QRSq + r_d^2 - CosTerm * r_d

to_polar(Vector, Levels)

barrel_vectordb_turboquant_subspace

Subspace-TurboQuant: O(D/M) scalable vector quantization

Addresses O(D^2) scaling issues in standard TurboQuant by splitting D-dimensional vectors into M independent subspaces. Each subspace applies TurboQuant with its own rotation matrix, reducing:

- Rotation matrix memory: D^2 -> M * (D/M)^2 = D^2/M (8x reduction for M=8) - Encode latency: O(D^2) -> O(D^2/M) per subspace, parallelizable - No training required (preserves data-oblivious property)

Storage format: Header: <<Version:8, Bits:8, M:8, Flags:8>> Body: [SubspaceCode_1, ..., SubspaceCode_M] Each subspace code contains radii, angles, and QJL signs

Performance for D=768, M=8: - Rotation matrices: 8 * 96^2 * 8 = 590KB (vs 4.7MB) - Encode latency: ~0.5ms (vs ~3.6ms) - Recall: Within 2-3% of full TurboQuant

Summary

Types

distance_tables/0

tq_subspace_code/0

tq_subspace_config/0

Functions

batch_distance_nif(Tables, Codes)

Batch compute ADC distance using SIMD-accelerated NIF

batch_encode(Config, Vectors)

Batch encode multiple vectors

decode(Tq_subspace_config, _)

Decode Subspace-TurboQuant code back to approximate vector

distance(Tables, _)

Compute asymmetric distance using precomputed tables (pure Erlang)

distance_nif(Tables, Code)

Compute ADC distance using SIMD-accelerated NIF

encode(Tq_subspace_config, Vector)

Encode a vector using Subspace-TurboQuant Returns compact binary with header + M subspace codes

info(Tq_subspace_config)

Get configuration info

new(Options)

Create a new Subspace-TurboQuant configuration Options: bits - bits per component (default: 3, range: 2-4) dimension - vector dimension (required, must be even and divisible by m) m - number of subspaces (default: auto-selected based on dimension) seed - base random seed (default: 42)

precompute_tables(Tq_subspace_config, Query)

Precompute distance lookup tables for a query vector Returns M sets of tables (concatenated)

select_m(D)

Auto-select M based on dimension for optimal performance Keeps subdim around 64-128 for best SIMD performance

split_subvectors(Vec, M, SubDim)

Types

distance_tables/0

-type distance_tables() :: binary().

tq_subspace_code/0

-type tq_subspace_code() :: binary().

tq_subspace_config/0

-type tq_subspace_config() ::
          #tq_subspace_config{bits :: 2..4,
                              m :: pos_integer(),
                              dimension :: pos_integer(),
                              subdim :: pos_integer(),
                              subspace_configs :: [term()],
                              seeds :: [integer()]}.

Functions

batch_distance_nif(Tables, Codes)

-spec batch_distance_nif(distance_tables(), [tq_subspace_code()]) -> [float()].

Batch compute ADC distance using SIMD-accelerated NIF

batch_encode(Config, Vectors)

-spec batch_encode(tq_subspace_config(), [[float()]]) -> [tq_subspace_code()].

Batch encode multiple vectors

decode(Tq_subspace_config, _)

-spec decode(tq_subspace_config(), tq_subspace_code()) -> [float()].

Decode Subspace-TurboQuant code back to approximate vector

distance(Tables, _)

-spec distance(distance_tables(), tq_subspace_code()) -> float().

Compute asymmetric distance using precomputed tables (pure Erlang)

distance_nif(Tables, Code)

-spec distance_nif(distance_tables(), tq_subspace_code()) -> float().

Compute ADC distance using SIMD-accelerated NIF

encode(Tq_subspace_config, Vector)

-spec encode(tq_subspace_config(), [float()]) -> tq_subspace_code().

Encode a vector using Subspace-TurboQuant Returns compact binary with header + M subspace codes

info(Tq_subspace_config)

-spec info(tq_subspace_config()) -> map().

Get configuration info

new(Options)

-spec new(map()) -> {ok, tq_subspace_config()} | {error, term()}.

Create a new Subspace-TurboQuant configuration Options: bits - bits per component (default: 3, range: 2-4) dimension - vector dimension (required, must be even and divisible by m) m - number of subspaces (default: auto-selected based on dimension) seed - base random seed (default: 42)

precompute_tables(Tq_subspace_config, Query)

-spec precompute_tables(tq_subspace_config(), [float()]) -> distance_tables().

Precompute distance lookup tables for a query vector Returns M sets of tables (concatenated)

select_m(D)

-spec select_m(pos_integer()) -> pos_integer().

Auto-select M based on dimension for optimal performance Keeps subdim around 64-128 for best SIMD performance

split_subvectors(Vec, M, SubDim)

-spec split_subvectors([float()], pos_integer(), pos_integer()) -> [[float()]].