barrel_vectordb (barrel_vectordb v2.1.1)
View Sourcebarrel_vectordb - Erlang Vector Database
An Erlang library for storing and searching vectors. Supports optional embedding providers for text-to-vector conversion.
Quick Start (with embeddings)
%% Start a store with local Python embeddings
{ok, _} = barrel_vectordb:start_link(#{
name => my_store,
path => "/tmp/vectors",
embedder => {local, #{}} %% requires Python + sentence-transformers
}).
%% Add a document (embeds the text automatically)
ok = barrel_vectordb:add(my_store, <<"doc-1">>, <<"Hello world">>, #{}).
%% Search with text query
{ok, Results} = barrel_vectordb:search(my_store, <<"greetings">>, #{k => 5}).Quick Start (vector-only, no embedder)
%% Start a store without embedder
{ok, _} = barrel_vectordb:start_link(#{
name => my_store,
path => "/tmp/vectors",
dimensions => 768
}).
%% Add with pre-computed vector
ok = barrel_vectordb:add_vector(my_store, <<"doc-1">>, <<"Hello">>, #{}, Vector).
%% Search with vector query
{ok, Results} = barrel_vectordb:search_vector(my_store, QueryVector, #{k => 5}).Configuration
#{
name => atom() | binary(), %% Store name (required)
path => string(), %% RocksDB path
dimensions => pos_integer(), %% Vector dimensions (default: 768)
embedder => EmbedderConfig, %% Embedding provider (optional)
hnsw => HnswConfig %% HNSW index parameters
}Embedding Providers
Embedder is **explicit** - if not configured, only add_vector/5 and search_vector/3 work. Text-based operations return {error, embedder_not_configured}.
%% Local Python with sentence-transformers (CPU, no external calls)
embedder => {local, #{
python => "python3",
model => "BAAI/bge-base-en-v1.5"
}}
%% Ollama (local LLM server)
embedder => {ollama, #{
url => <<"http://localhost:11434">>,
model => <<"nomic-embed-text">>
}}
%% Provider chain with fallback
embedder => [
{ollama, #{url => <<"http://localhost:11434">>}},
{local, #{}} %% Fallback to CPU
]Summary
Types
DiskANN index parameters.
Embedding provider configuration.
FAISS index parameters.
HNSW index parameters.
Unique document identifier.
Arbitrary metadata map associated with a document.
A single search result.
A store reference - a registered name (binary preferred; atoms are normalized, so dynamic names should be binaries) or a pid.
Document text content.
Embedding vector (list of floats).
Functions
Add a document with automatic embedding.
Add a document with explicit vector.
Add multiple documents in batch.
Index a vector without storing text or metadata.
Index multiple vectors without storing text or metadata.
Add a document with pre-computed vector (alias).
Add multiple documents with pre-computed vectors in batch.
Checkpoint the HNSW index to disk.
Get document count.
Delete a document.
Destroy a store: close every handle (RocksDB, index, disk BM25) and remove the storage directory. The store stops.
Generate embedding for a single text.
Generate embeddings for multiple texts.
Get embedding provider information.
Get a document by ID.
Peek at documents.
Search for similar documents using text query.
Search for similar documents using BM25 text search.
Hybrid search combining BM25 and vector search.
Search for similar documents using a vector query.
Start a new vector store.
Get store statistics.
Stop a vector store.
Update a document.
Insert or update a document.
Types
-type diskann_config() :: #{base_path => string() | binary(), l => pos_integer(), r => pos_integer()}.
DiskANN index parameters.
-type embedder_config() :: {local, map()} | {ollama, map()} | {openai, map()} | {anthropic, map()} | [{atom(), map()}].
Embedding provider configuration.
-type faiss_config() :: #{index_type => binary(), nprobe => pos_integer()}.
FAISS index parameters.
-type hnsw_config() :: #{m => pos_integer(), ef_construction => pos_integer(), distance_fn => cosine | euclidean}.
HNSW index parameters.
-type id() :: binary().
Unique document identifier.
Arbitrary metadata map associated with a document.
-type search_opts() :: #{k => pos_integer(), filter => fun((metadata()) -> boolean()), include_text => boolean(), include_metadata => boolean(), ef_search => pos_integer(), query_vector => [number()], bm25_weight => float(), vector_weight => float(), fusion => rrf | linear}.
-type search_result() :: #{key := id(), text := text(), metadata := metadata(), score := float(), vector => vector()}.
A single search result.
A store reference - a registered name (binary preferred; atoms are normalized, so dynamic names should be binaries) or a pid.
-type store_config() :: #{name := atom() | binary(), path => string() | binary(), db_path => string() | binary(), dimension => pos_integer(), dimensions => pos_integer(), embedder => embedder_config(), backend => hnsw | faiss | diskann, hnsw => hnsw_config(), faiss => faiss_config(), diskann => diskann_config(), bm25_backend => memory | disk | none, bm25 => map(), bm25_disk => map(), docstore => {module(), map()}, _ => _}.
-type text() :: binary().
Document text content.
-type vector() :: [float()].
Embedding vector (list of floats).
Functions
Add a document with automatic embedding.
Embeds the text using the configured provider and stores it in the vector database along with its metadata.
Example
ok = barrel_vectordb:add(my_store, <<"doc-1">>, <<"Hello world">>, #{
type => greeting,
language => english
}).Add a document with explicit vector.
Stores the document with a pre-computed embedding vector instead of generating one automatically.
Example
Vector = [0.1, 0.2, ...], %% 768 dimensions
ok = barrel_vectordb:add(my_store, <<"doc-1">>, <<"Hello">>, #{}, Vector).-spec add_batch(store(), [{id(), text(), metadata()}]) -> {ok, #{inserted := non_neg_integer()}} | {error, term()}.
Add multiple documents in batch.
Efficiently adds multiple documents with automatic embedding. More efficient than calling add/4 multiple times.
Example
Docs = [
{<<"id-1">>, <<"text 1">>, #{type => a}},
{<<"id-2">>, <<"text 2">>, #{type => b}}
],
{ok, #{inserted := 2}} = barrel_vectordb:add_batch(my_store, Docs).Index a vector without storing text or metadata.
For callers that own document storage elsewhere (for example a document database fronted by a barrel_vectordb_docstore adapter): the vector is written to the vector column family in the atomic batch (it stays authoritative for index rebuild on restart), Text feeds the BM25 index transiently and is then dropped, and no text/metadata is stored anywhere. Any stale text/metadata rows from a previous full add are cleared in the same batch.
Idempotent upsert by id: re-adding replaces the vector and the BM25 entry, so a crashed producer can safely re-drive the same entries. Use delete/2 to remove an entry.
-spec add_index_only_batch(store(), [{id(), text(), vector()}]) -> {ok, #{inserted := non_neg_integer()}} | {error, term()}.
Index multiple vectors without storing text or metadata.
Batch form of add_index_only/4: one atomic RocksDB write for all vectors.
Add a document with pre-computed vector (alias).
Same as add/5, provided for API clarity.
See also: add/5.
-spec add_vector_batch(store(), [{id(), text(), metadata(), vector()}]) -> {ok, #{inserted := non_neg_integer()}} | {error, term()}.
Add multiple documents with pre-computed vectors in batch.
Efficiently adds multiple documents with their vectors in a single atomic RocksDB write. Much faster than calling add_vector/5 multiple times.
Example
Docs = [
{<<"id-1">>, <<"text 1">>, #{type => a}, Vector1},
{<<"id-2">>, <<"text 2">>, #{type => b}, Vector2}
],
{ok, #{inserted := 2}} = barrel_vectordb:add_vector_batch(my_store, Docs).-spec checkpoint(store()) -> ok.
Checkpoint the HNSW index to disk.
Persists the current in-memory HNSW index metadata to RocksDB. This can speed up restart by avoiding a full index rebuild.
Note: The index is automatically rebuilt from vectors on startup if no checkpoint exists, so this is optional but improves restart time.
Example
ok = barrel_vectordb:checkpoint(my_store).-spec count(store()) -> non_neg_integer().
Get document count.
Returns the number of documents in the store.
Delete a document.
Removes a document from the store, including its vector and metadata.
Example
ok = barrel_vectordb:delete(my_store, <<"doc-1">>).Destroy a store: close every handle (RocksDB, index, disk BM25) and remove the storage directory. The store stops.
Generate embedding for a single text.
Uses the store's configured embedding provider to generate a vector. Useful when you need the vector for other purposes.
Example
{ok, Vector} = barrel_vectordb:embed(my_store, <<"hello world">>),
768 = length(Vector).Generate embeddings for multiple texts.
More efficient than calling embed/2 multiple times as it batches the requests to the embedding provider.
Example
{ok, Vectors} = barrel_vectordb:embed_batch(my_store, [<<"text 1">>, <<"text 2">>]),
2 = length(Vectors).Get embedding provider information.
Returns information about the configured embedding providers.
Example
{ok, Info} = barrel_vectordb:embedder_info(my_store),
Dimension = maps:get(dimension, Info),
Providers = maps:get(providers, Info).Get a document by ID.
Retrieves a document with its vector, text, and metadata.
Example
{ok, Doc} = barrel_vectordb:get(my_store, <<"doc-1">>),
Text = maps:get(text, Doc),
Meta = maps:get(metadata, Doc).-spec peek(store(), pos_integer()) -> {ok, [map()]}.
Peek at documents.
Returns a sample of documents without performing a search. Useful for inspecting the store contents.
Example
{ok, Docs} = barrel_vectordb:peek(my_store, 10),
[#{key := K, text := T, metadata := M} | _] = Docs.-spec search(store(), text(), search_opts()) -> {ok, [search_result()]} | {error, term()}.
Search for similar documents using text query.
Embeds the query text and finds the most similar documents using approximate nearest neighbor search (HNSW).
Example
{ok, Results} = barrel_vectordb:search(my_store, <<"hello">>, #{
k => 5,
filter => fun(Meta) -> maps:get(type, Meta, undefined) =:= greeting end
}),
[#{key := Key, text := Text, score := Score} | _] = Results.Search for similar documents using BM25 text search.
Performs keyword-based BM25 text search. Requires BM25 backend to be enabled in the store configuration.
Example
{ok, Results} = barrel_vectordb:search_bm25(my_store, <<"erlang programming">>, #{
k => 10
}),
[{DocId, Score} | _] = Results.-spec search_hybrid(store(), text(), search_opts()) -> {ok, [search_result()]} | {error, term()}.
Hybrid search combining BM25 and vector search.
Performs both BM25 text search and vector similarity search, then combines the results using a fusion algorithm (RRF or linear).
Example
{ok, Results} = barrel_vectordb:search_hybrid(my_store, <<"erlang">>, #{
k => 10,
bm25_weight => 0.5,
vector_weight => 0.5,
fusion => rrf %% or 'linear'
}).Results carry text and metadata like vector search (disable with include_text => false / include_metadata => false). Passing query_vector => Vector skips the internal embedder for the vector leg, so stores without an embedder can run hybrid search when the caller embeds the query itself.
-spec search_vector(store(), vector(), search_opts()) -> {ok, [search_result()]} | {error, term()}.
Search for similar documents using a vector query.
Finds the most similar documents to the given vector without needing to embed a text query first.
Example
QueryVector = [0.1, 0.2, ...], %% 768 dimensions
{ok, Results} = barrel_vectordb:search_vector(my_store, QueryVector, #{k => 5}).-spec start_link(store_config()) -> {ok, pid()} | {error, term()}.
Start a new vector store.
Creates a new vector store with the given configuration. The store is registered under the name provided in the config.
Example
{ok, Pid} = barrel_vectordb:start_link(#{
name => my_store,
path => "/var/lib/myapp/vectors",
dimensions => 768,
embedder => {local, #{}}
}).Get store statistics.
Returns information about the store including document count, dimensions, and HNSW index statistics.
Example
{ok, Stats} = barrel_vectordb:stats(my_store),
Count = maps:get(count, Stats),
Dims = maps:get(dimension, Stats).-spec stop(store()) -> ok.
Stop a vector store.
Gracefully shuts down the store, persisting any pending data.
Update a document.
Updates an existing document by re-embedding the text and storing the new text and metadata. Returns not_found if the document does not exist.
Example
ok = barrel_vectordb:update(my_store, <<"doc-1">>, <<"New text">>, #{updated => true}).Insert or update a document.
If the document exists, updates it. If not, inserts it. Always succeeds (unless there's an error).
Example
ok = barrel_vectordb:upsert(my_store, <<"doc-1">>, <<"Text">>, #{}).