barrel
The embeddable database. One barrel database is a document database and a vector store under one name and one id space, so a document, its attachments and its vector share an id.
What it solves
Agents and search features need documents, files and vectors that stay together. Keeping a document store and a vector index side by side means two ids, two lifecycles and writes that drift apart. barrel composes them, adds record mode (embeddings kept in sync with the documents), BQL with search table functions and live queries, timeline (branch, recover, merge), contexts and export.
When to use it
- You build an Erlang or Elixir application and want documents, attachments and vector search in-process.
- You want embeddings computed from document fields and kept in sync on every write (record mode).
- You want to branch a database for an agent run, review it, and merge it back.
When not to
- You need clients over the network: run barrel_server on top.
- You only need documents and sync, no vectors: barrel_docdb alone is enough.
- You need replication through this facade: it is not exposed here yet; call
barrel_repon the docdb name.
How it fits into Barrel
- Builds on
barrel_docdb,barrel_vectordb,barrel_embed,barrel_crypto- Used by
barrel_spaces,barrel_server- Other dependencies
- hackney (remote context client); OTP 26 or later
- On its own
- Yes, as a library. It serves no network protocol on its own.
Install
{barrel, "~> 1.14"} Record mode needs the barrel application started; otherwise opens with a policy return {error, barrel_app_not_started}.
Configuration
barrel:open/2 options
| Key | Default | What it does |
|---|---|---|
docdb | #{} | Options passed to barrel_docdb:create_db/2 |
vectordb | a vector store | Store config map, or none for documents only |
vectordb.db_path | <data_dir>/<name>.vectordb | Next to the documents. A store created before 1.13.0 in ./priv/barrel_vectordb_<name> stays there. |
embedding | none | Record-mode policy map, or stored to reuse the saved one |
encryption | disabled | default or #{provider => Mod} |
read_only | false | Writes return {error, read_only} |
store_supervised | false | By default the vector store is linked to the caller |
Record-mode policy
| Key | Default | What it does |
|---|---|---|
fields | all | Paths whose text is embedded |
mode | async | sync embeds before the write returns |
dimensions | 768 | Vector size |
embedder | none | A barrel_embed provider config |
metadata_fields | document minus id and _ keys | Fields copied into vector metadata |
App env (barrel)
| Key | Default | What it does |
|---|---|---|
dbs_idle_timeout | 300000 ms | barrel_dbs closes idle databases; 0 disables |
dbs_max_open | 0 | Open database limit; 0 is unlimited |
ctx_max_contexts | 8 | Contexts per federated query |
ctx_deadline_ms | 5000 | Context query deadline (max 60000) |
The most used keys only. For every option, see the configuration reference and the API reference.
The API by task
Open and manage
| Task | Call | Returns |
|---|---|---|
| Open or create | barrel:open/1,2 | {ok, Db} |
| Close, delete | barrel:close/1, delete/1 | ok |
| Describe | barrel:info/1 | docdb info plus embedding, dimensions, embedder |
| Managed lifecycle | barrel_dbs:ensure/1,2, lease/2, release/1 | {ok, Db}, {ok, Db, LeaseRef} |
Documents and files
| Task | Call | Returns |
|---|---|---|
| Write | barrel:put_doc/2,3, put_docs/2,3 | {ok, #{<<"id">>, <<"rev">>, <<"ok">>}} |
| Read | barrel:get_doc/2,3, get_docs/2,3 | {ok, Doc} |
| Delete | barrel:delete_doc/2,3 | {ok, _} |
| Find | barrel:find/2,3 | {ok, Rows, Meta} |
| Attach a file | barrel:put_attachment/4,5, get_attachment/3 | {ok, Info}, {ok, Bytes} |
Search and query
| Task | Call | Returns |
|---|---|---|
| Add a vector | barrel:vector_add/4,5, vector_add_batch/2 | ok, {ok, #{inserted}} |
| Semantic search | barrel:search/3, search_vector/3 | {ok, [#{key, score, text, metadata}]} |
| Keyword, hybrid | barrel:search_bm25/3, search_hybrid/3 | {ok, [{Id, Score}]}, {ok, [Hit]} |
| BQL | barrel:query/2,3, explain_query/2,3 | {ok, Rows, Meta} |
| Live query | barrel:subscribe_query/2,3, unsubscribe_query/1 | {ok, #{pid, ref}}, then bql_rows, bql_ready, bql_change messages |
Timeline and history
| Task | Call | Returns |
|---|---|---|
| Branch | barrel:branch/2,3 | {ok, BranchDb} |
| Merge back | barrel:merge/1,2 | {ok, Report} |
| Changes | barrel:changes/2,3, subscribe/2,3 | changes list, or a stream pid |
| Versions | barrel:doc_versions/2, version_body/3 | {ok, [Version]}, {ok, Body} |
Documents, vectors, BM25 and BQL in one database
From a verified run. kb_embed is a 64-dimension demo provider written for the run (feature hashing: it matches shared words, it is not semantic). Any barrel_embed provider works in its place.
{ok, _} = application:ensure_all_started(barrel).
{ok, Db} = barrel:open(<<"research">>, #{vectordb => #{
db_path => filename:join(VecDir, "research"),
dimensions => 64,
embedder => {kb_embed, #{}},
bm25_backend => memory}}).
Notes = [
{<<"paper:raft">>, <<"Raft consensus: leader election "
"and log replication for replicated state machines">>,
#{<<"topic">> => <<"distributed-systems">>,
<<"year">> => 2014}},
{<<"paper:crdt">>, <<"Conflict-free replicated data "
"types converge without coordination">>,
#{<<"topic">> => <<"distributed-systems">>,
<<"year">> => 2011}},
{<<"paper:hnsw">>, <<"HNSW: hierarchical navigable small "
"world graphs for approximate nearest neighbor "
"search">>,
#{<<"topic">> => <<"vector-search">>,
<<"year">> => 2016}},
{<<"paper:bm25">>, <<"Okapi BM25 ranking function for "
"keyword search over documents">>,
#{<<"topic">> => <<"information-retrieval">>,
<<"year">> => 1994}}].
[{ok, _} = barrel:put_doc(Db,
M#{<<"id">> => Id, <<"abstract">> => T})
|| {Id, T, M} <- Notes].
barrel:vector_add_batch(Db, Notes).
barrel:search(Db, <<"leader election in replicated logs">>,
#{k => 2}).
barrel:search_bm25(Db, <<"keyword search">>, #{k => 2}).
barrel:query(Db, <<"SELECT v.id, v._score, v._distance, "
"abstract FROM vector_top_k("
"'replicated state machines', k => 2) AS v">>). {ok,#{inserted => 4}}
{ok,[#{metadata => #{<<"topic">> => <<"distributed-systems">>,
<<"year">> => 2014},
text => <<"Raft consensus: ...">>,
key => <<"paper:raft">>,score => 0.5257737534501699},
#{metadata => #{<<"topic">> => <<"distributed-systems">>,
<<"year">> => 2011},
text => <<"Conflict-free replicated data types ...">>,
key => <<"paper:crdt">>,score => 0.14173227881546058}]}
{ok,[{<<"paper:bm25">>,1.9587592623731485},
{<<"paper:hnsw">>,0.6586049622596158}]}
{ok,[#{<<"_distance">> => 0.5158329928408527,
<<"_score">> => 0.4841670071591473,
<<"abstract">> => <<"Raft consensus: ...">>,
<<"id">> => <<"paper:raft">>},
#{<<"_distance">> => 0.7817821172596275,
<<"_score">> => 0.21821788274037246,
<<"abstract">> => <<"Conflict-free replicated ...">>,
<<"id">> => <<"paper:crdt">>}],
#{count => 2, has_more => false, ...}} Limits to know
- A vector store created before barrel 1.13.0 without
db_pathstays in./priv/barrel_vectordb_<name>under the current directory, with a notice in the log. A backup ofdata_diralone misses it; new stores go to<data_dir>/<name>.vectordb. - The vector store is linked to the process that opened it. Use
barrel_dbsorstore_supervised => truefor long-lived owners. - In plain mode,
put_docandvector_addare separate writes. Record mode is the atomic path. - Changing a record-mode policy on reopen logs a warning and does not reindex existing documents.
- Plain branches carry no vectors. A branch of a branch is refused (
cannot_branch_a_branch). - Export needs a vector store (
{error, no_vector_store}otherwise) and closes the database during the copy. - Context queries need
LIMIT; OFFSET, UNNEST, SUBSCRIBE and continuation are refused.
find/2 order_by on a path is ignored unless the path is selected or bound to a variable. BQL ORDER BY sorts correctly.