Record mode: policy-driven vector indexing
Record mode makes a document and its vector one unit: you write documents, and barrel keeps the vector index in sync with them, driven by a per-database embedding policy. The vector store holds only vectors and indexes; search results read text and metadata from the current documents. Read this when you want semantic or hybrid search over your documents without managing vectors yourself.
When to use it
- You write JSON documents and want vector/BM25/hybrid search over chosen fields, kept consistent with every update and delete.
- You do not want to call
vector_addby hand or duplicate text into a vector store. - For direct vector management (you own ids and vectors separately from
documents), use a plain database and the
vector_*API instead.
Open a record-mode database
Pass an embedding policy to barrel:open/2. The barrel application must
be running (it supervises the per-database indexer).
{ok, _} = application:ensure_all_started(barrel),
{ok, Db} = barrel:open(notes, #{
embedding => #{
fields => [<<"title">>, [<<"body">>, <<"text">>]],
mode => async, %% default; sync = read-your-write
embedder => {local, #{}}, %% barrel_embed provider chain
dimensions => 768,
metadata_fields => [<<"kind">>] %% optional projection
}
}).
fieldsare paths into the document; their binary values are joined (join, default<<"\n">>) into the text that embeds.- Without
metadata_fields, search metadata is the document minusidand_-prefixed keys. - BM25 defaults to the disk backend in record mode so keyword search survives restarts.
- The policy is persisted in the database; reopening with a different policy logs a warning and applies it to new writes only (no automatic reindex).
Write documents, search them
{ok, _} = barrel:put_doc(Db, #{<<"id">> => <<"a">>,
<<"title">> => <<"quick brown fox">>,
<<"kind">> => <<"animal">>}),
{ok, Hits} = barrel:search(Db, <<"fast fox">>, #{k => 5}),
{ok, HHits} = barrel:search_hybrid(Db, <<"fox">>, #{k => 5}),
{ok, BHits} = barrel:search_bm25(Db, <<"fox">>, #{k => 5}).
Hits carry key, score, and the document-derived text and metadata.
Updates re-embed; deletes remove the vector; documents without policy fields
get no vector.
To embed text with the policy's embedder yourself (for a search_vector/3
query, or to store the vector elsewhere), use the handle rather than its
fields:
{ok, Vector} = barrel:embed(Db, <<"fast fox">>),
{ok, Vectors} = barrel:embed_batch(Db, [<<"a">>, <<"b">>]),
{ok, #{dimensions := Dim}} = barrel:embedder_info(Db).
Check that two databases embed alike
embedder_info/1 also returns the embedder's identity, and info/1 carries
the same map under embedder. Two databases with the same fingerprint
produce comparable vector scores: a context query merges vector_top_k hits
by score only when every database reports the same fingerprint and cosine
distance (see contexts).
{ok, Db} = barrel:open(notes, #{embedding => #{
fields => [<<"title">>], dimensions => 768,
embedder => {ollama, #{model => <<"nomic-embed-text">>}}}}),
{ok, Info} = barrel:embedder_info(Db).
%% #{provider => ollama, model => <<"nomic-embed-text:latest">>,
%% dimensions => 768, distance => cosine,
%% preprocessing => #{fields => [[<<"title">>]], join => <<"\n">>},
%% fingerprint => <<"sha256:1b7105d2466397b07e625dbd608d33f5e76ecdf03c4f37aabaf8caa905165890">>,
%% ...}
fingerprintissha256:of the canonical JSON of provider, model,revision(when the provider reports one), dimensions, distance and preprocessing. Every provider of a fallback chain is part of it.- No fingerprint without a configured embedder, or when a provider does not
name its model. A plain database reports
preprocessing => none.
Async and sync
mode => async (default): writes return immediately and a supervised
indexer embeds in the background. The write and its indexing intent commit
atomically, so a crash at any point is healed by the indexer; no write is
ever lost between the document and its vector.
mode => sync: the text embeds before the write and the vector is indexed
before put_doc returns, so a search issued right after sees the document.
An embed failure fails the put with {error, {embed_failed, Reason}} and
nothing is written.
The _embedding property
Every indexed document's vector lives in the reserved _embedding property,
whoever computed it. It is stored as derived data alongside the document
(never in the body, so it does not change the revision), it is never
path-indexed, and it never appears in search metadata.
Supply your own embedding by carrying it in the document, as a bare vector or as an object; it is indexed instead of anything the policy would embed:
{ok, _} = barrel:put_doc(Db, #{<<"id">> => <<"a">>,
<<"title">> => <<"quick fox">>,
<<"_embedding">> => Vector}).
With client-supplied embeddings the policy can be empty: open with
embedding => #{dimensions => 768} (no fields) and no embedder is needed
at all. The vector length is checked against the database dimension before
the write; batches with a wrong-length _embedding are rejected whole.
Policy-computed vectors are stored in _embedding too: atomically with the
write in sync mode, via the indexer in async mode. Read the property back
with include_embedding; it always returns the object form, which carries
its provenance:
{ok, Doc} = barrel:get_doc(Db, <<"a">>, #{include_embedding => true}),
#{<<"vector">> := Vector, <<"source">> := Source} = maps:get(<<"_embedding">>, Doc),
%% Source is <<"client">> or <<"computed">>
Because the source travels inside the object, a read-modify-write that
resends a computed _embedding never freezes a stale vector: the policy
re-embeds when the text changes. Only vectors marked (or shaped as) client
input override the policy.
The vector put option is shorthand for supplying a client vector on one
write:
{ok, _} = barrel:put_doc(Db, Doc, #{vector => Vector}).
Precedence when several sources are present: the vector option, then a
client _embedding carried in the document, then the policy's fields.
Notes
vector_addandvector_add_batchreturn{error, record_mode}on record databases: the document is the only write path.- Embed failures in async mode retry per document; after 5 failures the document is parked (logged, its indexing entry stays pending and visible) and the rest of the queue keeps moving.
barrel:info/1reports the active policy and dimension.- Databases on the same local model (
local,fastembed,clip,splade,colbert: same model, Python and virtualenv) share one model process; it stops when the last of them closes. The process handles requests from all of them concurrently. - On
barrel_server, set theopen_optsapp env ofbarrel_serverto open every database with a policy, for example{barrel_server, [{open_opts, #{embedding => ...}}]}insys.config.